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Glossary 


4Rs — The  4Rs  Program  (Reading  Writing  Respect,  and  Resolution j 

ABC — Academic  and  Behavioral  Competencies  Program 

ADHD — attention  deficit  hyperactivity  disorder 

CDC — Centers  for  Disease  Control  and  Prevention 

combined-program — all  seven  programs  together 

CR — Child  Report 

CSP — Competence  Support  Program 

DIR — Decision  Information  Resources,  Inc.,  a subcontractor  to  Mathematica  Policy  Research,  Inc.  (MPR), 
the  contractor  selected  to  do  the  independent  evaluation  of  the  SACD  programs 

ES — effect  size 

HHS — U.S.  Department  of  Health  and  Human  Services 
HLM — hierarchical  linear  model 
ICC — intraclass  correlation 

IES — Institute  of  Education  Sciences,  U.S.  Department  of  Education 
LBW — Tove  In  a Big  World 

leavers — students  who  left  schools  that  were  participating  in  the  study  before  the  study  was  completed 
MANOVA — multivariate  analysis  of  variance 
MDES — minimum  detectable  effect  size 

middle  childhood — the  developmental  period  from  6 to  12  years  of  age 

MPR — Mathematica  Policy  Research,  Inc.,  the  contractor  selected  to  do  the  independent  evaluation  of  the 
SACD  programs 

multiprogram  evaluation — the  evaluation  of  the  impact  of  the  seven  SACD  programs  on  students’  social  and 
character  development.  The  evaluation  looked  at  the  average  impact  of  all  seven  SACD  programs  combined 
and  the  average  impact  of  each  SACD  program  individually. 

named  activities — activities  carried  out  in  the  context  of  a named  SACD  program  to  support  students’  social 
and  character  development 

named  program — a specific  SACD  program 

new  entrants — students  who  entered  schools  participating  in  the  study  after  the  study  had  begun 

outcome  domain — a set  of  outcomes  grouped  together  because  they  measure  a similar  school  activity,  teacher 
instructional  behavior,  or  student  behavior 
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outcomes — what  the  SACD  programs  were  expected  to  change  regarding  school  activities,  teacher 
instructional  behavior,  or  student  behavior 

PA — Positive  Action 

PATHS — Promoting  Alternative  Thinking  Strategies 

PCR — Primary  Caregiver  Report 

SACD — social  and  character  development 

SACD  activities — activities  to  support  students’  social  and  character  development 

SACD  goals — six  specific  goals  defined  by  the  SACD  Research  Program  and  promoted  through  the  use  of 
SACD  activities  to  support  students’  social  and  character  development.  The  goals  are  (1)  violence  prevention 
and  peace  promotion,  (2)  social  and  emotional  development,  (3)  character  education,  (4)  tolerance  and 
diversity,  (5)  risk  prevention  and  health  promotion,  and  (6)  civic  responsibility  and  community  service. 

SACD  program — a specific  program  that  has  the  purpose  of  improving  the  social  and  character  development 
of  students 

SACD  Research  Consortium — the  researchers  responsible  for  the  SACD  Research  Program.  These  include 
staff  at  the  Institute  of  Education  Sciences  (IES)  in  the  U.S.  Department  of  Education;  the  Division  of 
Violence  Prevention  in  the  National  Center  for  Injury  Prevention  and  Control,  Centers  for  Disease  Control 
and  Prevention  (CDC);  and  Mathematica  Policy  Research,  Inc.  (MPR),  the  contractor,  selected  to  do  the 
independent  evaluation  of  the  SACD  programs;  as  well  as  the  seven  research  teams  that  received  funding 
under  cooperative  agreements  to  evaluate  one  SACD  program  of  their  choosing  under  an  experimental 
design. 

SACD  Research  Program — a collaboration  between  the  Institute  of  Education  Sciences  (IES)  in  the  U.S. 
Department  of  Education  and  the  Division  of  Violence  Prevention  in  the  National  Center  for  Injury 
Prevention  and  Control,  Centers  for  Disease  Control  and  Prevention  (CDC).  Its  purpose  is  to  evaluate 
multiple  universal,  elementary  school-based  programs  that  target  student  social  development  and  behavior 
outcomes. 

SACD  training — professional  development  for  teachers  on  instructional  methods  to  promote  student  social 
and  character  development 

school-based  program — a program  delivered  in  a school  setting 

site — the  set  of  schools,  both  treatment  and  control,  specific  to  each  individual  SACD  program  evaluation 
SS — Second  Step 

stayers — students  who  were  in  schools  participating  in  the  study  for  the  entire  study 

targeted  school-based  programs — programs  that  are  designed  to  address  the  needs  of  a subset  of  the  children 
in  a school 

TRCS — Teacher  Report  on  Classroom  and  School 
TRS — Teacher  Report  on  Student 

universal  programs — programs  that  are  designed  to  address  the  needs  of  all  children  in  a school 
WWC— the  What  Works  Clearinghouse 
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Executive  Summary 


A variety  of  universal  school-based  programs  designed  to  help  elementary  schools  foster  positive  student 
behaviors,  reduce  negative  behaviors,  and,  ultimately,  improve  academic  performance  are  available;  however, 
more  evidence  from  rigorous  evaluations  is  needed  to  better  understand  their  effects.  Such  information  is 
important  because  the  development  of  social  competencies  during  middle  childhood  has  been  linked  to 
adjustment  to  schooling  and  academic  success,  while  the  failure  to  develop  such  competencies  can  lead  to 
problem  behavior  that  interferes  with  success  in  school  (Bennett  et  al.  2003;  Carlson  et  al.  1999;  Farrington 
1989;  Fors,  Crepaz,  and  Flayes  1999;  Malecki  and  Elliot  2002;  McCord  et  al.  2000;  Najaka,  Gottfredson,  and 
Wilson  2001;  O’Donnell,  Hawkins,  and  Abbott  1995;  Trzesniewski  et  al.  2006;  Wentzel  1993). 1 

The  Institute  of  Education  Sciences  (IES)  and  the  Division  of  Violence  Prevention  in  the  National  Center  for 
Injury  Prevention  and  Control,  Centers  for  Disease  Control  and  Prevention  (CDC)  collaborated  to  conduct  a 
rigorous  impact  evaluation  of  programs  aimed  at  improving  students’  behavior.  For  this  evaluation,  such 
programs  were  termed  Social  and  Character  Development  (SACD)  programs.2  Seven  programs  were 
evaluated,  and  all  were  coherent  in  that  their  activities  were  integrated  and  logically  organized  based  on  a 
theory  of  action  (that  differed  among  the  programs),  school-based  in  that  they  were  implemented  in  the 
schools  by  school  personnel,  and  universal  in  that  they  were  to  be  implemented  for  all  students  in  all 
elementary  classrooms  in  a school. 

This  report  provides  the  results  from  the  evaluation  of  the  seven  SACD  programs  on  one  cohort  of  students 
as  they  moved  from  third  through  fifth  grades  starting  in  fall  2004  and  ending  in  spring  2007. 3 The  evaluation 
examined  the  effects  on  these  students  of  the  seven  programs,  together  and  separately,  after  1,  2,  and  3 
school  years  and  also  estimated  the  impact  on  students’  growth  in  social  and  character  development  over  the 
3 years.  Chapter  1 discusses  the  evaluation  of  the  programs  when  considered  together  and  provides  summary 
results  for  each  program.  Chapters  2 through  8 detail  the  findings  for  each  of  the  programs  individually. 
There  are  two  appendixes:  appendix  A examines  whether  the  addition  of  the  smaller  second  cohort  of 
students  to  the  study  affected  the  results,  and  appendix  B contains  additional  technical  information 
concerning  the  analyses. 


1 More  information  on  the  value  of,  evidence  for,  and  theories  behind  these  programs  can  be  found  in  the  second  and 
third  sections  of  chapter  1. 

2 Activities  carried  out  by  these  programs  in  support  of  students’  social  and  character  development  are  termed  SACD 
activities.  The  SACD  evaluation  examined  SACD  activities  intended  to  promote  six  goals  (termed  SACD  goals)  and 
behavior  management.  The  six  SACD  goals  included  (1)  violence  prevention  and  peace  promotion,  (2)  social  and 
emotional  development,  (3)  character  education,  (4)  tolerance  and  diversity,  (5)  risk  prevention  and  health  promotion, 
and  (6)  civic  responsibility  and  community  service. 

3 From  2005  to  2007,  a smaller,  second  cohort  of  students  was  followed  from  third  through  fourth  grades  in  a separate 
set  of  schools.  This  cohort  is  discussed  in  appendix  A.  The  Executive  Summary  and  chapters  1 through  8 describe  the 
analysis  of  Cohort  1 only. 
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Study  Design 

Through  a competitive  application  process  that  included  a scientific  peer  review,  seven  research  teams 
received  funding  under  cooperative  agreements  to  evaluate  one  SACD  program  of  their  choosing  under  an 
experimental  design.  Each  research  team  recruited  10  to  14  schools4  (for  a total  of  84),  with  half  of  the 
schools  implementing  one  of  the  seven  SACD  programs  for  the  3 years  of  the  study  (the  treatment  group) 
and  the  other  half  continuing  with  their  traditional  SACD  activities  (the  control  group).  Each  team’s  schools 
were  randomly  assigned  to  the  treatment  or  control  group  through  a stratified  sampling  process.  All  but  one 
school  (a  control  school)  participated  in  the  study  for  the  full  3 years.  Table  A lists  the  research  teams,  the 
SACD  program  each  evaluated,  key  features  of  each  program,  and  the  number  of  treatment  and  control 
schools.5 

Under  a separate  peer-reviewed  competition,  Mathematica  Policy  Research,  Inc.  (MPR)  received  a contract  to 
carry  out  a multiprogram  evaluation  of  the  seven  SACD  programs  using  (1)  standardized  data  collection  for 
all  sites,  (2)  a common  set  of  descriptive  measures  on  the  types  and  level  of  school-based  activities  (i.e., 
SACD  activities)  that  targeted  social  and  behavioral  outcomes  at  both  the  treatment  and  control  schools,  (3)  a 
common  set  of  outcome  measures,  and  (4)  a uniform  statistical  analysis.6  The  evaluation  examined  the 
impacts  of  all  seven  programs  together,  assessed  the  effect  of  each  program  separately  to  identify  any 
contrasts  with  the  overall  findings  across  programs,  and  tested  differences  in  effects  on  student  subgroups. 

Data  were  first  collected  from  students,  their  primary  caregivers,  teachers,  and  principals  in  the  fall  of  2004 
when  students  were  starting  third  grade.  Follow-up  data  were  collected  at  four  time  points:  (1)  end  of  third 
grade  (spring  2005),  (2)  beginning  of  fourth  grade  (fall  2005),  (3)  end  of  fourth  grade  (spring  2006),  and  (4) 
end  of  fifth  grade  (spring  2007).  Some  students  stayed  at  the  schools  in  the  study  for  the  full  3 years  (stayers), 
others  left  (leavers)  and  were  not  followed,  and  new  students  entered  the  schools  in  the  first  (after  the  initial 
data  collection),  second,  or  third  years  of  the  study  (new  entrants).  All  students  in  each  grade  of  the  cohort 
were  included  in  the  sample.  Table  B describes  the  student  sample  overall  and  by  program  for  all  students 
and  for  the  treatment  and  control  groups.  The  study  began  with  about  6,600  students  in  third  grade  and 
ended  with  about  6,200  in  fifth  grade.  By  the  end  of  the  fifth  grade,  31  percent  of  the  original  sample  had  left 
and  new  entrants  made  up  28  percent  of  the  fifth -graders.  There  were  no  statistically  significant  differences  in 
the  percentages  of  leavers  and  new  entrants  between  the  treatment  and  control  groups  overall,  though  there 
were  some  such  differences  within  the  individual  programs.7 


4 New  York  University  recruited  18  schools  but  only  14  were  included  in  this  evaluation. 

5 A longer  version  of  table  A with  more  detail  on  the  programs  appears  as  table  1 in  chapter  1 . Chapters  2 through  8 
provide  greater  discussion  of  each  program. 

6 Research  teams  carried  out  program-specific  evaluations  using  their  own  procedures  and  measures.  These  are  to  be 
published  separately  in  the  literature. 

7 Greater  detail  on  the  experimental  design  and  on  the  sample  is  provided  in  chapter  1 under  Study  Design  and 
Methodology. 


xxvt 


Executive  Summary 


Table  A.  Research  teams,  SACD  programs,  and  number  of  schools 


Research  team 

Program 

Program  features 

Number  of 
Treatment 
schools 

Number  of 
Control 
schools 

University  at  Buffalo, 
State  University  of 
New  York 

Academic  and 
Behavioral 
Competencies 
Program 

Social  skills  training  and  behavior 
management 

6 

6 

University  of  North 
Carolina  at  Chapel 
Hill 

Competence  Support 
Program 

Social  and  emotional  learning, 
social  dynamics  training,  and 
behavior  management:  social 
information  processing,  social 
problem  solving,  peer  networks 

5 

5 

Vanderbilt  University 

Love  In  a Big  World 

Character  education:  courage, 
honesty,  kindness,  caring 

6 

6’ 

Oregon  State 
University 

Positive  Action 

Social  and  emotional  learning: 
values,  empathy,  self-control, 
social  skills,  social  bonding,  self- 
efficacy,  honesty,  goal  setting 

7 

7 

The  Children’s 
Institute 

Promoting  Alternative 
Thinking  Strategies 

Social  and  emotional  learning: 
emotional  literacy,  self-control, 
social  competence,  peer 
relations,  interpersonal  problem 
solving 

5 

5 

New  York  University 

The  4Rs  Program 
(Reading,  Writing, 
Respect,  and 
Resolution) 

Conflict  resolution  and  literacy: 
social  problem  solving,  anger 
management,  mediation 

7 

7 

University  of 
Maryland,  College 
Park 

Second  Step 

Violence  prevention  and  social 
and  emotional  learning:  empathy, 
anger  management,  impulse 
control,  and  problem  solving 

6 

6 

1 Dropped  to  five  after  one  control  school  became  a magnet  school  and  dropped  out  of  the  study  prior  to  Year  2. 
SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Table  B. 


Student  sample,  overall  and  by  program,  for  all  students  and  for  the  treatment  and  control  groups 


Year  1 Year  1 Year  2 Year  3 

(Fall  3rd  grade)  (Spring  3rd  grade) (Spring  4th  grade) (Spring  5th  grade) 

New  New  New 


Intervention  program 

All 

students 

All 

students 

Leavers 

entrants 

All 

students 

Leavers 

entrants 

All 

students 

Leavers 

entrants 

# 

“/o1 

# 

%2 

# 

°/o 

# 

%2 

# 

°/o 

# 

%2 

All  programs 

6,567 

6,597 

364 

6 

394 

6 

6,415 

1,457 

22 

1,305 

20 

6,249 

2,067 

31 

1,749 

28 

Treatment  group 

3,367 

3,388 

179 

5 

200 

6 

3,327 

742 

22 

702 

21 

3,172 

1,078 

32 

883 

28 

Control  group 

3,200 

3,209 

185 

6 

194 

6 

3,088 

715 

22 

603 

20 

3,077 

989 

31 

866 

28 

ABC 

879 

875 

43 

5 

39 

4 

877 

160 

18 

158 

18 

871 

289 

33 

281 

32 

Treatment  group 

380 

373 

17 

4 

10 

3*. 

367 

72 

19 

59 

16 

353 

135 

36 

108 

31 

Control  group 

499 

502 

26 

5 

29 

6 

510 

88 

18 

99 

19 

518 

154 

31 

173 

33 

CSP 

959 

975 

36 

4 

52 

5 

969 

230 

24 

240 

25 

947 

238 

25 

226 

24 

Treatment  group 

476 

485 

20 

4 

29 

6 

474 

135 

28** 

133 

28* 

458 

139 

29*. 

121 

26 

Control  group 

483 

490 

16 

3 

23 

5 

495 

95 

20 

107 

22 

489 

99 

20 

105 

21 

LBW 

986 

1,007 

60 

6 

81 

8 

959 

228 

23 

201 

21 

944 

308 

31 

266 

28 

Treatment  group 

548 

565 

25 

5** 

42 

7 

556 

110 

20 

118 

21 

567 

145 

26** 

164 

29 

Control  group 

438 

442 

35 

8 

39 

9 

403 

118 

27 

83 

21 

377 

163 

37 

102 

27 

PA 

811 

812 

74 

9 

75 

9 

764 

251 

31 

204 

27 

655 

408 

50 

252 

38 

Treatment  group 

410 

416 

33 

8 

39 

9 

425 

108 

26** 

123 

29 

327 

209 

51 

126 

39 

Control  group 

401 

396 

41 

10 

36 

9 

339 

143 

36 

81 

24 

328 

199 

50 

126 

38 

PATHS 

786 

783 

39 

5 

36 

5 

778 

150 

19 

142 

18 

778 

243 

31 

235 

30 

Treatment  group 

377 

374 

21 

6 

18 

5 

373 

66 

18 

62 

17 

378 

114 

30 

115 

30 

Control  group 

409 

409 

18 

4 

18 

4 

405 

84 

21 

80 

20 

400 

129 

32 

120 

30 

4Rs 

1,202 

1,194 

86 

7 

78 

7 

1,109 

320 

27 

227 

20 

1,065 

492 

41 

355 

33 

Treatment  group 

652 

647 

49 

8 

44 

7 

599 

183 

28 

130 

22 

556 

279 

43 

183 

33 

Control  group 

550 

547 

37 

7 

34 

6 

510 

137 

25 

97 

19 

509 

213 

39 

172 

34 

See  notes  at  end  of  table. 
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Table  B.  Student  sample,  overall  and  by  program,  for  all  students  and  for  the  treatment  and  control  groups — Continued 


Intervention  program 

Year  1 

(Fall  3rd  grade) 

Year  1 

(Spring  3rd  grade) 

Year  2 

(Spring  4th  grade) 

Year  3 

(Spring  5th  grade) 

All 

students 

All 

students 

Leavers 

New 

entrants 

All  . 
students 

Leavers 

New 

entrants 

All  _ 

Leavers 

New 

entrants 

# “/o1 

# 

%2 

# Yc1 

# %2 

students 

# “/o1 

# %2 

SS 

944 

951 

26  3 

33 

3 

959 

118  13 

133  14 

989 

89  9 

134  14 

Treatment  group 

524 

528 

14  3 

18 

3 

533 

68  13 

77  14 

533 

57  11* 

66  12 

Control  group 

420 

423 

12  3 

15 

4 

426 

50  12 

56  13 

456 

32  8 

68  15 

* Treatment  group  significantly  different  from  control  group  at  the  .05  level. 

**  Treatment  group  significantly  different  from  control  group  at  the  .01  level. 

1 Leavers  as  a percentage  of  fall  2004  enrollment  (these  values  are  cumulative  over  the  years). 

2 New  entrants  as  a percentage  of  spring  enrollment. 

NOTE:  Abbreviations  are 

ABC:  Academic  and  Behavioral  Competencies  Program 
CSP:  Competence  Support  Program 
LBW:  Love  in  a Big  World 
PA:  Positive  Action 

PATHS:  Promoting  Alternative  Thinking  Strategies 

4Rs:  The  4Rs  Program  (Reading,  Writing,  Respect,  and  Resolution) 

SS:  Second  Step 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Twenty  student  and  school  outcomes  related  to  social  and  character  development  were  used  to  evaluate  the 
impact  of  the  SACD  programs  on  student  outcomes  and  perceptions  of  school  climate.8  These  were  grouped 
into  four  domains:  (1)  Social  and  Emotional  Competence,  which  contained  three  outcomes;  (2)  Behavior, 
which  contained  nine  outcomes;  (3)  Academics,  which  contained  two  outcomes;9  and  (4)  Perceptions  of 
School  Climate,  which  contained  six  outcomes.  Four  major  data  collection  instruments  were  used  to  collect 
the  scales  on  which  the  outcome  measures  were  based.  For  the  Child  Report,  students  individually  completed 
a set  of  10  scales  during  school,  and  these  contributed  to  10  outcomes  that  fell  across  all  four  domains.  For 
the  Primary  Caregiver  Report,  primary  caregivers  (usually  parents)  filled  in  a written  survey  (or  completed  a 
phone  interview  if  they  failed  to  complete  the  written  version)  that  contained  12  scales.  Six  of  these 
contributed  to  three  outcomes  in  the  Behavior  domain  (the  other  6 were  used  to  develop  measures  of 
characteristics  associated  with  greater  prevalence  of  child  behavior  problems).  In  the  Teacher  Report  on 
Student,  each  student’s  teacher  reported  on  10  scales  regarding  a student’s  typical  behavior  in  the  past  30 
days,  and  these  were  converted  into  five  outcomes  in  the  Behavior  and  Academics  domains.  In  the  Teacher 
Report  on  Classroom  and  School,  the  third-,  fourth-,  and  fifth-grade  teachers  in  a school  reported  on  two 
scales  that  were  converted  into  two  outcomes  that  described  their  Perceptions  of  School  Climate.  In  addition, 
they  completed  items  that  described  the  level  of  SACD  activity  in  the  classroom  and  school.  Table  C lists  the 
20  outcomes  by  domain  and  the  reports  from  which  they  were  obtained.  Three  of  the  outcomes  in  the 
Behavior  domain  were  measured  in  more  than  one  report.  Altruistic  Behavior  and  Problem  Behavior  were 
separately  measured  three  times  using  responses  from  children,  primary  caregivers,  and  teachers;  Positive 
Social  Behavior  was  separately  measured  twice  using  responses  from  primary  caregivers  and  teachers. 


8 The  original  scales  and  the  outcomes  are  described  in  chapter  1 under  Measures. 

9 The  Academics  domain  was  planned  to  but  did  not  include  student  grades  and  standardized  test  scores;  not  all  research 
teams  were  able  to  collect  these  data  from  their  schools  and  districts,  and  the  data  that  were  collected  varied  in  quality. 
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Table  C.  Outcomes,  by  domain  and  data  instrument 


Domain/Outcome 

Instrument 

Child  Report 

Primary 

Caregiver 

Report 

Teacher  Report 
on  Student 

Teacher  Report 
on  Classroom 
and  School 

Social  and  Emotional  Competence  Domain 

Self-Efficacy  for  Peer  Interaction  (+) 

V 

Normative  Beliefs  About  Aggression  (-) 

V 

Empathy  (+) 

V 

Behavior  Domain 

Altruistic  Behavior  (+) 

y' 

Positive  Social  Behavior  (+) 

y' 

Problem  Behavior  (-) 

✓ 

ADHD-Related  Behavior  (-) 

■/ 

Academics  Domain 

Engagement  with  Learning  (+) 

Academic  Competence  and  Motivation  (+) 

✓ 

Perceptions  of  School  Climate  Domain 

Positive  School  Orientation  (+) 

Negative  School  Orientation  (-) 

Student  Afraid  at  School  (-) 

Victimization  at  School  (-) 

Feelings  of  Safety  (+) 

Student  Support  for  Teachers  (+) 

■/ 

NOTE:  Abbreviations  are 
ADHD:  Attention  deficit  hyperactivity  disorder 
s : Outcome  addressed 
Blank  cell:  Outcome  not  addressed 

The  +/-  signs  in  parentheses  indicate  the  direction  of  a beneficial  outcome. 
SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Research  Questions 

The  purpose  of  the  SACD  evaluation  was  to  determine  whether  seven  coherent,  universal,  school-based 
programs  improved  student  social  and  emotional  competence;  improved  behavior,  including  reducing 
negative  behavior;  improved  student  achievement;  and  improved  student  and  teacher  perceptions  of  school 
climate.  The  evaluation  considered  the  programs  together  and  individually.  In  addition,  the  evaluation 
considered  the  programs’  impacts  on  students  with  different  backgrounds  that  have  been  found  to  increase 
the  risk  of  poor  outcomes  and  possibly  change  student  responses  to  the  SACD  programs.  Also,  the 
evaluation  took  into  account  findings  from  previous  work  showing  that  the  level  of  implementation  of  a 
program  affects  its  impact.  In  addition,  the  expectation  of  a positive  impact  on  student  outcomes  raised  the 
issue  of  whether  and  in  what  ways  the  programs  increased  the  prevalence  of  SACD  instruction.  These  issues 
led  to  the  development  of  five  research  questions: 

1 . What  is  the  average  effect  of  the  seven  universal,  school-based,  social  and  character  development 
programs  on  social  and  character  development  instruction  in  the  schools? 

2.  What  is  the  average  effect  of  the  seven  universal,  school-based,  social  and  character  development 
programs  on  students’  social  and  emotional  competence,  behavior,  and  academics,  and  on 
perceptions  of  school  climate? 

3.  What  is  the  average  effect  of  each  specific  social  and  character  development  program  on  students’ 
social  and  emotional  competence,  behavior,  and  academics,  and  on  perceptions  of  school  climate? 

4.  Do  the  average  effects  of  the  seven  universal,  school-based  social  and  character  development 
programs  differ  by  (a)  students’  gender  and  (b)  students’  initial  risk  factors  (socioeconomic,  family, 
community,  and  earlier  child  behavior)? 

5.  In  the  treatment  schools,  is  there  an  association  between  the  level  of  implementation  of  the  social 
and  character  development  programs  and  student  outcomes? 

Data  Collection 

Three  issues  regarding  the  conduct  of  the  data  collection  have  implications  for  the  analysis:  (1)  timing  of  data 
collection,  (2)  percentages  of  the  sample  for  which  data  were  available  for  analysis,  and  (3)  students  who  left 
the  study  (leavers). 

Ideally,  the  first  fall  data  collection  would  have  started  at  the  very  beginning  of  the  year  to  reduce  the 
possibility  that  program  implementation  could  have  affected  responses  to  the  student,  primary  caregiver,  and 
teacher  surveys.  For  practical  reasons,  fall  data  collection  was  often  delayed  for  several  weeks  to  allow  school 
populations  to  settle,  to  obtain  primary  caregiver  consent,  and  to  avoid  disrupting  planned  school  activities. 
As  a result,  program  implementation  began  before  initial  data  collection  for  six  of  the  research  teams 
(Vanderbilt  University  was  the  exception).  This  interval  ranged  from  2 to  6 weeks.  In  addition,  at  all  schools 
teachers  and  principals  received  training  on  the  intervention  before  the  fall  2004  data  collection.  As  a result, 
the  fall  2004  reports  from  teachers  and  principals,  and  possibly  students,  are  unlikely  to  reflect  the  true  pre- 
intervention condition  but  instead  capture  what  was  being  done  at  the  beginning  of  the  evaluation. 10 

Data  were  not  successfully  collected  from  all  students,  primary  caregivers,  and  teachers.  Data  were  not 
collected  when  written  consent  was  not  provided  by  primary  caregivers  or  teachers,  or  when  respondents 
refused  to  take  part  (even  after  consent  had  been  given)  or  were  unavailable  at  the  time  of  data  collection. 
Table  D presents  the  overall  consent  and  completion  rates  for  each  report  by  year  and  by  treatment  versus 
control  group.  This  table  also  presents  the  percentages  of  the  sample  for  which  there  are  data  for  each  report. 
These  are  calculated  by  multiplying  the  consent  rate  by  the  completion  rate.  Table  D shows  that  60  percent  to 


10  Additional  details  on  timing  issues  can  be  found  under  Data  Collection  in  chapter  1 . 
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65  percent  of  students  had  data  supplied  by  themselves  over  the  3 years,  46  percent  to  59  percent  had  data 
provided  by  their  primary  caregivers,  and  61  percent  to  67  percent  had  data  provided  by  teachers.  In  Year  1 
(third  grade),  a statistically  significant  larger  percentage  of  the  treatment  group  had  data  than  the  control 
group;  there  were  no  significant  differences  in  Years  2 and  3.  Table  D also  shows  that  data  on  classrooms  and 
schools  were  obtained  from  86  percent  to  90  percent  of  the  teachers.11 

The  evaluation  did  not  follow  all  students  originally  assigned  to  the  treatment  or  control  groups  (what  is 
known  as  an  “intent  to  treat”  study)  because  data  were  not  collected  from  students  who  left  the  schools.  If 
the  SACD  programs  caused  differential  student  mobility  in  the  treatment  schools  versus  the  control  schools, 
then  the  impact  of  this  mobility  would  be  combined  with  the  impact  of  the  programs  and  the  two  could  not 
be  disentangled.  Descriptive  analyses,  presented  in  chapter  1,  did  not  identify  statistically  significant 
differential  mobility  in  the  treatment  and  control  schools,  but  this  is  not  definitive  evidence  that  it  does  not 
exist. 


11  For  additional  comparisons  see  Consent  Rates  in  chapter  1 and  chapters  2 through  8 regarding  each  program. 
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Consent  rates,  completion  rates,  and  percentage  of  sample  with  data  from  each  report 


Year  1 Year  1 Year  2 Year  3 

(Fall  3rd  grade)  (Spring  3rd  grade)  (Spring  4th  grade)  (Spring  5th  grade) 

Treat-  Treat-  Treat-  Treat- 


Report 

Total 

ment 

Control 

Total 

ment 

Control 

Total 

ment 

Control 

Total 

ment 

Control 

Student  sample  size 

6,567 

3,367 

3,200 

6,597 

3,388 

3,209 

6,415 

3,327 

3,088 

6,249 

3,172 

3,077 

Child  Report  (percent) 

Primary  caregiver  consent  rate 

65 

67** 

63 

66 

68** 

64 

67 

67 

66 

66 

67 

66 

Student  completion  rate 

94 

93* 

94 

96 

96 

96 

95 

96 

95 

96 

97 

96 

Students  with  data1 

61 

62* 

60 

63 

65** 

61 

63 

65 

62 

64 

65 

63 

Primary  Caregiver  Report  (percent) 

Primary  caregiver  consent  rate 

63 

64** 

61 

64 

66** 

62 

64 

65 

63 

64 

65 

64 

Primary  caregiver  completion  rate 

92 

92 

92 

80 

80 

81 

78 

78 

77 

72 

71 

72 

Primary  caregivers  with  data1 

57 

59* 

56 

51 

52 

50 

50 

51 

49 

46 

46 

46 

Teacher  Report  on  Student  (percent) 

Primary  caregiver  consent  rate2 

65 

67** 

63 

66 

68** 

64 

67 

67 

66 

66 

67 

66 

Teacher  completion  rate 

96 

96 

96 

99 

99 

99 

100 

100** 

99 

98 

98 

99 

Students  with  data1 

62 

64** 

61 

65 

67** 

63 

66 

67 

65 

65 

66 

65 

Teacher  Report  on  Classroom  and 
School  (3rd-  to  5th-grade  teachers) 
(percent) 

Teacher  consent  rate 

96 

98*** 

92 

95 

97* 

94 

95 

97 

94 

96 

97 

95 

Teacher  completion  rate 

91 

90 

93 

91 

90 

91 

94 

94 

94 

92 

91 

93 

Teachers  with  data1 

87 

88 

86 

87 

88 

86 

90 

90 

89 

89 

88 

89 

* Treatment  group  significantly  different  than  control  group  at  the  .05  level. 

**  Treatment  group  significantly  different  than  control  group  at  the  .01  level. 

***  Treatment  group  significantly  different  than  control  group  at  the  .001  level. 

1 Calculated  as  consent  rate  x completion  rate. 

2 The  primary  caregiver  consent  rates  for  the  Child  Report  and  the  Teacher  Report  on  Student  are  identical,  as  the  primary  caregiver  gave  consent  to  both  together. 
SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Initial  Characteristics 

An  examination  of  the  initial  characteristics  of  the  students,  families,  teachers,  and  schools  found  that  the 
treatment  and  control  groups  were  similar  on  a set  of  observed  characteristics  (with  the  exception  of  the  use 
of  SACD  activities  in  the  schools),  providing  evidence  that  the  random  assignment  of  schools  within 
programs  created  similar  groups.12  The  data  for  this  examination  were  collected  in  fall  2004  from  enrolled 
third-grade  students,  their  primary  caregivers,  and  their  third-grade  teachers.  In  addition,  third-,  fourth-,  and 
fifth-grade  teachers  and  principals  in  the  study  schools  provided  information  about  SACD  activities  being 
used  in  the  classroom  and  school.  The  initial  data  were  collected  after  the  staff  at  the  treatment  schools  began 
receiving  training  in  their  programs  and,  for  six  of  the  programs,  after  implementation  had  begun. 

The  sample’s  treatment  and  control  groups  were  similar  along  the  observed  student,  primary  caregiver,  and 
community  characteristics.  These  included  (1)  student  gender,  race/ ethnicity,  and  age;  (2)  primary  caregiver 
race/ethnicity,  age,  employment,  marital  status,  education,  and  household  income;  and  (3)  community  risks 
and  resources.  In  addition,  there  were  no  significant  differences  between  the  treatment  and  control  groups  for 
the  20  outcome  measures  and  the  five  measures  of  initial  risk,  showing  that  the  two  groups  of  students 
started,  on  average,  at  the  same  place  in  third  grade. 

The  teachers  in  the  treatment  and  control  schools  were  similar  in  gender,  race/ ethnicity,  years  of  teaching, 
and  certification.  The  only  statistically  significant  difference  concerned  degree  attainment,  as  a larger 
percentage  of  treatment  teachers  (60%)  had  a master’s  or  doctoral  degree  than  did  control  teachers  (52%). 
There  was  no  significant  difference  between  the  treatment  and  control  schools  with  regard  to  student 
composition  (race/ ethnicity  and  school  lunch  eligibility),  number  of  students,  number  of  full-time  teachers, 
Title  I status,  highest  and  lowest  grades,  urbanicity,  and  number  of  years  the  principal  had  been  at  the  school. 
There  was  also  no  significant  difference  in  teacher  reports  on  nine  dimensions  concerning  their  school 
environment:  feelings  of  safety,  resource  adequacy,  student  support,  staff  freedom  to  teach  as  desired, 
affiliation  with  and  ties  to  colleagues,  innovation  regarding  trying  new  approaches  to  teaching,  professional 
interest,  participatory  decisionmaking,  and  work  pressure. 

An  examination  of  the  initial  level  of  activities  to  promote  social  and  character  development  in  the  classroom 
and  schoolwide,  and  of  the  materials  and  methods  used  in  these  activities,  revealed  that  the  control  condition 
for  the  evaluation  was  not  a “no  treatment”  control.  Rather,  it  was  a “standard  practice”  control  condition,  in 
which  more  than  half  of  teachers  and  80  percent  to  90  percent  of  principals  reported  schoolwide  and 
classroom  activities  designed  to  promote  social  and  character  development.  Standard  practice  at  the  control 
schools  included  the  use  of  specific  materials  and  practices  to  promote  social  and  character  development,  as 
well  as  professional  development  related  to  social  and  character  development  of  students  for  staff.  Many 
kinds  of  activities  and  strategies  were  provided  at  rates  and  in  types  and  amounts  similar  to  those  reported  in 
treatment  schools. 

Treatment  teachers  reported  greater  use  of  and  training  in  SACD  activities  than  control  teachers  more  often 
than  would  be  expected  by  chance.  This  may  reflect  actual  differences  in  the  use  of  SACD  activities  prior  to 
implementation  of  the  programs,  or  it  may  reflect  that  program  implementation  (for  six  programs)  and 
program  training  for  staff  had  started  before  initial  data  collection.13 


12  For  greater  detail,  see  Initial  Characteristics  in  chapter  1. 

13  For  details  on  these  comparisons,  see  The  Initial  Level  of  Social  and  Character  Development  in  the  Schools  in 
chapter  1 . 
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Analysis  and  Results 

Four  sets  of  analyses  were  done  to  evaluate  the  impacts  of  the  SACD  programs.14  First,  the  effects  of  the 
adoption  of  the  programs  on  the  use  of  SACD  activities  in  the  classroom  and  school  were  examined  on  an 
annual  basis  for  Years  1 to  3.  Second,  the  programs’  impacts  on  student  outcomes  and  perceptions  of  school 
climate  were  analyzed  at  the  end  of  each  of  the  3 years  of  the  study  (i.e.,  from  the  start  of  third  grade  to  the 
end  of  third  grade,  to  the  end  of  fourth  grade,  and  to  the  end  of  fifth  grade).  Third,  subgroup  analyses  were 
done  on  the  same  annual  basis  to  examine  whether  the  programs’  impacts  on  student  outcomes,  when 
combined,  differed  by  four  subgroups:  (1)  girls  versus  boys,  (2)  stayers  versus  new  entrants,  (3)  students  with 
different  levels  of  baseline  risk,  and  (4)  students  in  treatment  schools  with  high  versus  low  fidelity  of 
implementation.  Fourth,  a growth  curve  analysis  was  done  to  determine  impacts  on  annual  student  growth  in 
the  outcomes  over  the  3 years.  All  but  the  third  analysis  were  done  for  the  combination  of  all  seven  programs 
and  for  each  program  separately. 

Annual  Impacts  on  the  Use  of  Social  and  Character  Development  Activities 

The  SACD  programs  were  expected  to  increase  the  use  of  activities  to  promote  social  and  character 
development  in  the  classroom  and  school.  The  analysis  of  activity  use  was  based  on  data  from  the  Teacher 
Report  on  Classroom  and  School  (TRCS),  which  was  filled  out  by  all  the  third-,  fourth-,  and  fifth-grade 
teachers.  Data  from  the  TRCS  were  used  to  create  83  SACD  activity  outcome  measures,  and  these  were 
grouped  into  six  domains:  (1)  teacher  use  of  any  activities  to  promote  social  and  character  development, 
which  had  1 6 variables;  (2)  teacher  use  of  any  activities  associated  with  a specific  SACD  program  (known  as  a 
“named”  program),  which  had  14  variables;  (3)  teacher  use  of  materials  and  classroom  strategies  for  SACD 
activities,  which  had  29  variables;  (4)  use  of  schoolwide  strategies,  which  had  6 variables;  (5)  teacher 
involvement  in  related  professional  development,  which  had  9 variables;  and  (6)  teacher  support  for  SACD 
efforts  in  the  school  and  the  prevalence  of  an  environment  conducive  to  the  social  and  character 
development  of  students,  which  had  9 variables. 

To  estimate  the  impact  of  the  SACD  programs  on  use  of  SACD  activities,  the  statistical  significance  of  the 
differences  in  means  between  the  treatment  and  control  groups  was  tested  for  each  of  the  83  SACD  activity 
outcome  measures.15  For  the  overall  analysis,  the  data  were  first  weighted  to  give  equal  weight  to  each 
program  and  to  each  school  within  a site.  For  the  program -by-program  analyses,  each  school  within  a site 
received  equal  weight  (school  weights  differed  between  program  analyses,  as  the  number  of  schools  was  not 
constant  among  programs). 

In  addition  to  estimating  the  impacts  of  the  SACD  programs  on  the  individual  outcome  measures,  the 
impacts  on  the  six  domains  were  also  examined.  Testing  the  impact  on  the  domains  was  done  to  adjust  for 
the  multiple  comparisons  made  within  each  domain  in  order  to  address  the  increased  chances  of  finding  a 
spurious  outcome  when  more  than  one  test  was  done.  As  a result,  two  sets  of  results  were  obtained:  (1)  the 


14  The  analytical  techniques  used  for  each  of  the  four  sets  of  analyses  are  summarized  in  the  following  sections  and 
discussed  in  detail  in  chapter  1 . 

15  Three  factors  contributed  to  the  decision  to  use  differences  in  means.  First,  because  of  random  assignment,  simple 
treatment-control  contrasts  provided  unbiased  estimates  of  program  impacts.  Second,  only  initial  values  (rather  than  true 
baseline  values)  for  these  outcomes  were  available  to  use  in  a model  because  training  (at  all  treatment  schools)  and 
program  implementation  (at  36  treatment  schools)  began  before  data  collection.  The  decision  to  use  initial  values  in  an 
analysis  partly  depends  on  whether  the  initial  training  and  implementation  occurring  before  data  collection  would  be 
expected  to  have  immediate  and  large  impacts  on  the  outcomes  (Schochet  2008b).  For  this  analysis,  the  outcomes  are 
based  on  teacher  actions  and  so  would  likely  be  upwardly  influenced  by  the  teacher  training  and  short  period  of  teacher 
implementation  before  pretesting  (in  contrast  to  student  outcomes,  which  would  be  less  likely  to  be  so  influenced).  For 
this  reason,  a model-based  analysis  using  the  initial  values  as  covariates  was  not  chosen.  Third,  preliminary  analyses 
indicated  no  gain  in  precision  from  the  inclusion  of  other  covariates. 
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impacts  on  the  individual  outcomes  unadjusted  for  multiple  comparisons,  and  (2)  the  impacts  on  the  domains 
that  serve  as  the  multiple  comparison  adjustment.  To  test  whether  the  SACD  programs  had  a statistically 
significant  impact  on  each  of  the  six  domains,  a set  of  three  heuristics  was  used  and  a significant  effect  on  the 
domain  was  found  if  any  one  of  the  heuristics  was  met.16 

The  results  provide  evidence  that  the  SACD  programs  increased  the  reported  implementation  of  SACD 
activities  in  the  classroom.  Over  the  3 years,  249  comparisons  (83  outcome  variables  times  3 years)  of 
treatment  and  control  teachers  were  tested,  with  12  to  13  expected  to  be  statistically  significant  by  chance. 
The  analysis  found  127  comparisons  were  statistically  significant,  with  all  showing  greater  reported  use  of 
SACD  activities  by  treatment  teachers.  When  examining  the  individual  outcomes  within  each  domain,  the 
analysis  found  the  following:  (1)  treatment  teachers  reported  significantly  greater  use  of  SACD  activities  in 
the  classroom  for  3 1 of  the  48  comparisons,  (2)  treatment  teachers  reported  significandy  greater  use  of  SACD 
activities  linked  to  a named  SACD  program  for  39  of  the  42  comparisons,  (3)  treatment  teachers  reported 
significantly  greater  use  of  materials  and  instructional  methods  to  promote  social  and  character  development 
for  40  of  the  87  comparisons,  (4)  there  were  no  significant  differences  for  the  1 8 comparisons  made  regarding 
the  use  of  schoolwide  strategies,  (5)  treatment  teachers  reported  significantly  greater  receipt  of  training  to 
promote  social  and  character  development  for  13  of  the  27  comparisons,  and  (6)  treatment  teachers  reported 
significantly  greater  use  of  practices  conducive  to  social  and  character  development  but  similar  attitudes 
toward  it  for  4 of  the  27  comparisons.17 

The  results  from  the  analysis  of  the  individual  outcomes  also  provide  further  evidence  that  the  control  group 
was  a “standard  practice”  rather  than  a “no  treatment”  control.  Over  the  3 years,  control  teachers  continued 
to  report  use  of  SACD  activities.  For  example,  over  the  3 years,  86  percent  to  90  percent  of  control  teachers 
reported  using  a SACD  activity  to  address  any  one  of  the  six  SACD  goals  versus  95  percent  to  96  percent  of 
the  treatment  teachers.  Similarly,  over  the  3 years,  20  percent  to  36  percent  of  control  teachers  reported  using 
a SACD  activity  linked  to  a named  SACD  program  to  address  any  one  of  the  six  SACD  goals  versus  68 
percent  to  72  percent  of  the  treatment  teachers.18 

The  findings  from  the  analysis  of  the  six  domains  are  consistent  with  the  results  from  the  individual  outcome 
analysis.  For  all  3 years,  treatment  teachers  reported  statistically  significant  greater  implementation  in  four  of 
the  six  SACD  activity  domains:  (1)  use  of  any  SACD  activities  in  the  classroom,  (2)  use  of  SACD  activities 
from  named  programs  in  the  classroom,  (3)  use  of  materials  and  teaching  strategies  for  SACD  activities,  and 
(4)  participation  in  relevant  professional  development.  No  evidence  was  found  that  the  programs  affected  the 
other  two  domains — the  use  of  schoolwide  strategies  and  attitudes  and  practices  that  create  an  environment 
conducive  to  students’  social  and  character  development.  Table  E shows  where  statistically  significant  impacts 
on  the  use  of  SACD  activities  occurred  by  domain  for  all  seven  programs  and  for  each  program  by  year.  In 
table  E,  a plus  sign  indicates  a significant  positive  impact  on  the  domain,  and  superscript  numerals  show 
which  heuristics  identified  the  domain  as  significant. 


16  The  three  heuristics  included  (1)  determining  if  more  than  half  of  the  individual  outcomes  within  a domain  had  a 
similar  and  statistically  significant  impact,  (2)  doing  one  overall  test  of  impact  on  all  outcomes  within  a domain,  and 
(3)  checking  for  any  statistically  significant  results  among  the  outcomes  within  a domain  after  applying  a specific 
statistical  adjustment  for  multiple  comparisons  (Benjamini-Hochberg  1995)  to  each  one.  See  Year-by-Year  Impacts  on 
Use  of  Social  and  Character  Development  Activities  in  chapter  1 for  longer  descriptions  of  these  heuristics  and  how  they 
were  used. 

17  For  more  details  on  the  impacts  on  the  individual  outcomes  making  up  the  domains  see  Year-by-Year  Impacts  on  Use 
of  Social  and  Character  Development  Activities  in  chapter  1 . 

18  For  more  details  on  the  percent  of  treatment  and  control  teachers  reporting  on  the  use  of  SACD  activities  see  Year- 
by-Year  Impacts  on  Use  of  Social  and  Character  Development  Activities  in  chapter  1. 
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Table  E.  Significant  impacts  on  use  of  SACD  activity  domains,  overall  and  by  program 


SACD  activity  domain 

SACD  activities 

Classroom 

Attitudes 

SACD 

linked  to  named 

materials  and  Schoolwide 

Professional 

and 

Program 

activities 

programs 

strategies  strategies 

development 

practices 

Overall 

Year  1 

+1,2,3 

+1,2,3 

+2’3 

+1,2,3 

Year  2 

+1,2,3 

+1,2,3 

+1,2,3 

+2’3 

Year  3 

+1,2,3 

+1,2,3 

+1,2,3 

+2’3 

ABC 

Year  1 

2 

+ 

1 

+ 

Year  2 

Year  3 

+3 

CSP 

Year  1 

2 

+ 

+1'3 

+3 

Year  2 

+3 

Year  3 

+3 

LBW 

Year  1 

+2,3 

+3 

+3 

+3 

Year  2 

+3 

+3 

2 

+ 

Year  3 

PA 

Year  1 

+2,3 

+1'3 

+3 

+3 

Year  2 

+1'3 

Year  3 

+3 

+3 

PATHS 

Year  1 

+2’3 

+1'3 

+1'3 

Year  2 

+3 

+1'3 

Year  3 

+1'3 

4Rs 

Year  1 

+1,2,3 

+ 1,2,3 

+3 

+1'3 

Year  2 

+1'3 

+1'3 

Year  3 

+1'3 

+ 1,2,3 

+3 

+3 

See  notes  at  end  of  table. 
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Table  E.  Significant  impacts  on  use  of  SACD  activity  domains,  overall  and  by  program — 
Continued 


Program 

SACD  activity  domain 

SACD 

activities 

SACD  activities 
linked  to  named 
programs 

Classroom 
materials  and 
strategies 

Schoolwide 

strategies 

Professional 

development 

Attitudes 

and 

practices 

SS 

Year  1 

+3 

+1’3 

+3 

Year  2 

+u 

+1’3 

+3 

Year  3 

+3 

+1’3 

+3 

1 Based  on  univariate  statistical  tests,  at  least  half  of  the  impacts  were  positive  and  statistically  significant  and  no  impact  was 
negative  and  statistically  significant. 

2 The  omnibus  impact  for  all  the  outcomes  measured  together  was  positive  and  statistically  significant  on  the  basis  of  a multivariate 
statistical  test. 

3 At  least  one  outcome  remained  positive  and  statistically  significant  and  no  outcome  was  negative  and  statistically  significant  after 
applying  the  Benjamini-Hochberg  (1 995)  procedure  to  adjust  significance  levels  downward  to  account  for  the  multiple  testing  of 
impacts. 

NOTE:  Abbreviations  are 

ABC:  Academic  and  Behavioral  Competencies  Program 
CSP:  Competence  Support  Program 
LBW:  Love  In  a Big  World 
PA:  Positive  Action 

PATHS:  Promoting  Alternative  Thinking  Strategies 

4Rs:  The  4Rs  Program  (Reading,  Writing,  Respect,  and  Resolution) 

SS:  Second  Step 
Abbreviations  of  the  findings  are 

+:  Statistically  significant  beneficial  impact  on  domain 
Blank  cell:  Finding  of  no  statistically  significant  impact 

Significance  is  based  on  p < .05.  No  detrimental  impact  was  found  statistically  significant  at  or  below  the  .05  level.  Description  of 
SACD  Activity  Domains  and  the  heuristics  used  to  determine  the  statistically  significant  beneficial  impact  on  the  domain  (for  more 
detail,  see  the  Measures  section  in  chapter  1): 

SACD  activities:  based  on  16  teacher-reported  measures  on  the  use  of  SACD  activities  in  the  classroom. 

SACD  activities  linked  to  named  programs:  based  on  14  teacher-reported  measures  on  the  use  of  SACD  activities  associated 
with  a named  program  in  the  classroom. 

Classroom  materials  and  strategies:  based  on  29  teacher-reported  measures,  7 concerning  materials  used  in  the  classroom  and 
22  concerning  classroom  strategies. 

Schoolwide  strategies:  based  on  six  teacher-reported  measures  concerning  strategies  to  promote  SACD  schoolwide. 
Professional  development:  based  on  nine  teacher-reported  measures  concerning  their  participation  in  SACD-related  training. 
Attitudes  and  practices:  based  on  nine  teacher-reported  measures,  three  concerning  teacher  attitudes  toward  SACD  efforts  in 
the  school  and  six  concerning  school  practices  conducive  to  the  social  and  character  development  of  students. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 


Annual  Impacts  on  Student  Outcomes  and  Perceptions  of  School  Climate 

The  SACD  programs  were  expected  to  improve  children’s  social  and  emotional  competence,  behavior, 
academics,  and  perceptions  of  school  climate  as  measured  by  20  outcome  variables.  One  test  of  these 
hypotheses  was  to  examine  the  year-by-year  impacts  of  the  SACD  programs  on  these  outcomes  over  the  3 
years  as  the  students  progressed  from  third  through  fifth  grades.  The  examination  of  year-by-year  impacts 
entailed  three  sets  of  analyses  resulting  in  three  sets  of  impacts.  The  first  set  of  analyses  compared  the 
outcomes  of  treatment  and  control  students  from  the  fall  of  third  grade  to  the  spring  of  third  grade.  The 
second  set  compared  the  outcomes  from  the  fall  of  third  grade  to  the  spring  of  fourth  grade,  and  the  third  set 
compared  outcomes  from  the  fall  of  third  grade  to  the  spring  of  fifth  grade.  Within  each  set  of  year-by-year 
analyses,  an  analysis  of  all  the  programs  together  provided  impact  results  for  the  set  of  seven  coherent, 
universal,  school-based  programs,  and  separate  analyses  of  each  individual  program  provided  results  specific 
to  each  program.  The  combined  analysis  was  able  to  detect  smaller  statistically  significant  impacts,  because  of 
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its  larger  sample  size  and  the  associated  greater  power,  than  the  analyses  of  each  SACD  program,  which  were 
based  on  smaller  samples.19 

The  random  assignment  of  schools  ensured  that  unbiased  estimates  of  the  average  impacts  of  the  SACD 
programs  (relative  to  the  social  and  character  development  activities  offered  in  the  control  schools)  could  be 
computed  as  the  differences  in  the  average  outcomes  of  students  and  teachers  in  the  treatment  and  control 
schools.  However,  regression  procedures  were  used  rather  than  simple  differences -in-means  procedures  to 
estimate  impacts  to  improve  the  statistical  precision  of  the  estimates;  to  address  the  clustering  of  students 
within  schools;  and  to  adjust  for  differences  between  treatment  and  control  group  observable  characteristics 
due  to  random  selection,  study  nonconsent,  and  interview  nonresponse.  A hierarchical  linear  model  (HLM) 
was  used  to  estimate  regression- adjusted  impacts  (Bryk  and  Raudenbush  1992).  The  basic  model  consisted  of 
two  levels  that  were  indexed  by  students  or  teachers  and  by  schools.  The  model  included  covariates  that 
adjusted  for  statistically  significant  treatment  and  control  differences  at  initial  data  collection.  These  covariates 
were  chosen  because  they  had  predictive  power  across  a broad  range  of  outcomes.  Sample  weights  were  used 
in  the  analyses  in  order  to  (1)  give  each  site  equal  weight  in  the  calculation  of  pooled  impact  estimates,  (2)  give 
each  school  equal  weight  in  each  site,  and  (3)  adjust  for  missing  outcome  data  due  to  nonconsent  and 
nonresponse.  The  model  was  estimated  using  data  from  all  seven  programs  together  (using  all  three  types  of 
weights)  and  individually  for  each  program  (using  the  second  and  third  types  of  weights).  A set  of  analyses  to 
examine  differences  by  subgroup  for  the  combined  data  was  done  by  examining  the  significance  of  a 
coefficient  on  an  interaction  term  between  the  treatment  status  indicator  variable  and  subgroup  indicator 
variable  (or  multiple  coefficients  and  multiple  interaction  terms  when  there  were  more  than  two  subgroups) 
when  the  subgroup  variables  were  included.  The  association  of  fidelity  of  implementation  with  the  outcomes 
was  examined  in  a similar  way,  using  a fidelity  indicator  variable  and  an  interaction  term  between  the  fidelity 
variable  and  the  treatment  status  variable.20 

Results  from  estimating  the  model  were  provided  in  effect  sizes.21  A standard  two-tailed  test  was  used  to 
determine  the  y>-value  for  the  coefficient  of  each  outcome  measure.  Coefficients  with  ^-values  of  .05  or  below 
were  considered  statistically  significant  and  identified  as  such.  Impacts  that  were  not  statistically  significant 
but  were  .25  standard  deviation  units  or  more  in  magnitude  were  identified  as  “substantively  important,” 
following  the  practice  used  by  the  What  Works  Clearinghouse.22  Substantively  important  impacts  identify 
effects  that  may  be  large  enough  to  have  practical  importance  but  are  not  found  to  be  statistically  significant, 
potentially  because  of  sample  size  constraints. 

The  20  outcome  variables  were  grouped  under  a set  of  four  domains:  (1)  Social  and  Emotional  Competence, 
which  contained  3 outcomes;  (2)  Behavior,  which  contained  9 outcomes;  (3)  Academics,  which  contained  2 
outcomes;  and  (4)  Perceptions  of  School  Climate,  which  contained  6 outcomes.  As  in  the  case  of  the  SACD 
activities  domains,  a set  of  heuristics  was  used  to  test  the  significance  of  the  impacts  on  the  four  outcome 


19  The  combined  analysis  provided  a sample  size  sufficient  to  detect  student-level  impacts  at  minimum  detectable  effect 
sizes  ranging  from  0.03  to  0.24  standard  deviations  (see  table  1.25  in  chapter  1 for  details). 

20  For  a discussion  of  the  regression  model,  see  Analysis  under  the  section  Year-by-Year  Impacts  on  Use  of  Social  and 
Character  Development  Activities  in  chapter  1. 

21  Effect  sizes  were  calculated  by  dividing  the  estimated  impact  (the  coefficient  estimated  by  the  regression  model)  by  the 
standard  deviation  of  the  outcome  measure  for  the  control  group.  The  standard  deviation  was  calculated  using  data  for 
the  weighted  control  group.  It  was  calculated  at  the  time  of  data  collection  for  which  the  effect  size  (impact)  was 
estimated. 

22  The  What  Works  Clearinghouse  was  established  in  2002  by  the  U.S.  Department  of  Education’s  Institute  of 
Education  Sciences  to  provide  educators,  policymakers,  researchers,  and  the  public  with  a central  and  trusted  source  of 
scientific  evidence  of  what  works  in  education. 
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domains  to  adjust  for  the  multiple  comparisons  made  within  each  domain.23  The  analysis  of  the  year-by-year 
impacts  then  produced  two  sets  of  results:  (1)  the  impacts  on  the  individual  outcomes  unadjusted  for  multiple 
comparisons,  and  (2)  the  impacts  on  the  domains  that  served  as  the  multiple  comparison  adjustment. 

Column  2 of  table  F reports  the  number  of  statistically  significant  and  substantively  important  impacts  found 
for  the  year-by-year  analysis  of  all  seven  programs  together.  Specifically,  2 of  60  estimated  impacts  on  the 
outcomes  for  the  seven  SACD  programs  combined  for  the  3 years  were  found  statistically  significant  (versus 
3 expected  by  chance).  The  2 statistically  significant  impacts  were  that  the  combined  SACD  programs  had  a 
beneficial  impact  on  the  teacher-reported  measure  for  Student  Support  for  Teachers  in  Years  1 and  2,  with 
effect  sizes  of  0.12  and  0.16,  respectively.  None  of  the  remaining  58  estimated  impacts  were  found  to  be 
substantively  important.24  In  addition,  12  impacts  on  the  domains  were  estimated  (4  domains  times  3 years), 
with  1 expected  to  be  statistically  significant  by  chance.  The  analysis  found  2 significant  negative  impacts  on 
the  domain  of  Social  and  Emotional  Competence  in  Years  2 and  3 (these  data  are  not  shown  in  a table).  The 
results  for  the  individual  outcomes  and  the  domains  provide  no  evidence  that  the  SACD  programs  improved 
student  outcomes  and  perceptions  of  school  climate. 

The  lack  of  statistically  significant  impacts  found  in  the  combined-program  analysis  was  not  due  to  offsetting 
beneficial  and  detrimental  impacts  among  the  individual  programs.  For  the  individual  SACD  programs,  16 
significant  impacts  were  found  over  the  3 years  (9  beneficial  and  7 detrimental)  versus  21  expected  by  chance 
from  the  420  statistical  tests  done  (see  column  3 of  table  F).  In  only  one  program  did  a significant  impact 
occur  on  an  outcome  in  more  than  one  year,  and  in  this  case  the  impact  was  beneficial  in  Year  1 and 
detrimental  in  Year  2.  In  addition,  19  substantively  important  impacts  were  found  (10  beneficial  and  9 
detrimental).  There  was  little  replication  of  the  substantively  important  impacts:  one  program  had  a 
substantively  important  beneficial  impact  on  one  outcome  for  all  3 years.  Table  G identifies  the  statistically 
significant  and  nonsignificant  substantively  important  results  by  program,  outcome,  and  year.  It  provides  a 
visual  view  of  the  balance  between  beneficial  and  detrimental  impacts  and  of  the  lack  of  persistence  in 
impacts  by  program  across  the  years. 


23  The  heuristics  were  used  to  determine  whether  the  results  for  the  multiple  outcomes  within  each  domain  showed  a 
statistically  significant  impact  on  the  domain  as  a whole.  A significant  effect  on  the  domain  was  found  if  any  one  of  the 
heuristics  was  met.  In  addition  to  the  three  heuristics  mentioned  in  footnote  16,  a fourth  heuristic  (not  applicable  to  the 
analysis  of  teacher-reported  SACD  activities)  was  used.  For  this  heuristic,  the  statistical  model  used  to  estimate  impacts 
on  the  individual  outcomes  was  re-estimated  using  a composite  of  all  the  outcome  variables  within  a domain.  The 
composite  was  formed  by  standardizing  each  outcome  variable  using  its  standard  deviation,  combining  the  values  of  the 
outcome  variables,  and  taking  the  average  of  the  final  value. 

24  For  the  actual  effect  size  and  significance  level  of  each  outcome,  see  table  1.26  in  chapter  1. 


Executive  Summary 


Table  F.  Significant  impacts  on  student  outcomes,  overall  and  by  program,  for  all  3 years 


Domain  (number  of  outcomes) 

Combined-program  analysis 
(all  seven  programs  together) 
(60  impact  estimates)1 

Individual  program  analyses 
(each  program  separately) 
(420  impact  estimates)^ 

Total 

Statistically  significant  outcomes 

2 

16 

(Beneficial) 

(9  Beneficial,  7 Detrimental) 

Substantively  important  outcomes 

0 

19 

(10  Beneficial,  9 Detrimental) 

Social  and  Emotional  Competence  (3) 

Statistically  significant  outcomes 

0 

0 

Substantively  important  outcomes 

0 

0 

Behavior  (9) 

Statistically  significant  outcomes 

0 

8 

(6  Beneficial,  2 Detrimental) 

Substantively  important  outcomes 

0 

6 

(2  Beneficial,  4 Detrimental) 

Academics  (2) 

Statistically  significant  outcomes 

0 

4 

(1  Beneficial,  3 Detrimental) 

Substantively  important  outcomes 

0 

0 

Perceptions  of  School  Climate  (6) 

Statistically  significant  outcomes 

2 

4 

(Beneficial) 

(2  Beneficial,  2 Detrimental) 

Substantively  important  outcomes 

0 

13 

(8  Beneficial,  5 Detrimental) 

1 For  each  year,  20  impacts  (on  20  outcomes)  were  estimated  and  1 statistically  significant  impact  would  be  expected  each  year  by 
chance  (for  a total  of  3). 

2 For  each  year,  140  impacts  were  estimated  (7  programs  by  20  outcomes)  and  7 statistically  significant  impacts  would  be  expected 
each  year  by  chance  (for  a total  of  21). 

NOTE:  For  each  outcome,  a finding  of  “beneficial”  indicates  the  program(s)  had  a beneficial  impact  on  that  particular  outcome;  a 
finding  of  “detrimental”  indicates  a detrimental  impact  on  that  outcome.  Significance  is  based  on  p < .05.  The  number  of  results 
found  significant  was  no  more  than  expected  by  chance. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Table  G. 


Individual  program  statistically  significant  impacts  and  nonsignificant  but  substantively  important  impacts 


Statistically  significant1 Nonstatistically  significant  but  substantive2 


Beneficial  impacts 

Detrimental  impacts 

Beneficial  impacts 

Detrimental  impacts 

Program 

(Report)  (Effect  size)  (p-value) 

(Report)  (Effect  size)  (p-value) 

(Report)  (Effect  size)  (p-value) 

(Report)  (Effect  size)  (p-value) 

Total 

9 

7 

10 

9 

Year  1 

3 

2 

2 

0 

Year  2 

5 

2 

6 

1 

Year  3 

1 

3 

2 

8 

ABC 

Year  1 

Altruistic  Behavior 
(TRS)  (.39)  (.026) 

Year  2 

Academic  Competence 
(CR)  (.31)  (.011) 
Feelings  of  Safety 
(TRCS)  (.75)  (.003) 

Altruistic  Behavior 
(CR)  (-.20)  (0.029) 

Student  Support 
for  Teachers 
(TRCS)  (.27)  (.276) 

Year  3 

Positive  Social  Behavior 
(PCR)  (.21)  (.041) 

Feelings  of  Safety 
(TRCS)  (.31)  (.235) 

CSP 

Year  1 

Year  2 

Problem  Behavior 
(PCR)  (-.21)  (.042) 

Altruistic  Behavior 
(TRS)  (.47)  (.132) 
Student  Afraid  at  School 
(CR)  (-.26)  (.090) 

Year  3 

Altruistic  Behavior 
(TRS)  (-.41)  (.132) 
Feelings  of  Safety 
(TRCS)  (-.36)  (.246) 

LBW 

Year  1 

Altruistic  Behavior 
(PCR)  (.31)  (.005) 
Student  Support 
for  Teachers 
(TRCS)  (.52)  (.022) 

Year  2 

Student  Support 
for  Teachers 
(TRCS)  (.28)  (.428) 

Altruistic  Behavior 
(TRS)  (-.34)  (.270) 

See  notes  at  end  of  table. 
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Table  G.  Individual  program  statistically  significant  impacts  and  nonsignificant  but  substantively  important  impacts — Continued 

Statistically  significant1 Nonstatistically  significant  but  substantive2 

Beneficial  impacts  Detrimental  impacts  Beneficial  impacts  Detrimental  impacts 


Program (Report)  (Effect  size)  (p-value)  (Report)  (Effect  size)  (p-value)  (Report)  (Effect  size)  (p-value)  (Report)  (Effect  size)  (p-value) 


Year  3 

Engagement  with  Learning 
(CR)  (-.35)  (.030) 
Positive  School  Orientation 
(CR)  (-.33)  (.047) 
Feelings  of  Safety 
(TRCS)  (-.70)  (.046) 

Problem  Behavior 
(CR)  (.31)  (.223) 
Student  Support  for  Teachers 
(TRCS)  (-.26)  (.543) 

PA 

Year  1 

Engagement  with  Learning 
(CR)  (-.25)  (.01 7) 

Altruistic  Behavior 
(TRS)  (.27)  (.480) 

Year  2 

Positive  Social  Behavior 
(PCR)  (.24)  (.039) 
Problem  Behavior 
(TRS)  (-.24)  (.048) 

Student  Support  for  Teachers 
(TRCS)  (.28)  (.113) 

Year  3 

PATHS 

Year  1 

Year  2 

Year  3 

Altruistic  Behavior 
(TRS)  (-.31)  (.485) 
Feelings  of  Safety 
(TRS)  (-.29)  (.582) 

4Rs 

Year  1 

Academic  Competence 
(CR)  (-.17)  (.032) 

Year  2 

Year  3 

Feelings  of  Safety 
(TRS)  (-.42)  (.146) 
Student  Support  for  Teachers 
(TRCS)  (-.35)  (.109) 

See  notes  at  end  of  table. 
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Table  G.  Individual  program  statistically  significant  impacts  and  nonsignificant  but  substantively  important  impacts — Continued 


Statistically  significant1 Nonstatistically  significant  but  substantive2 


Beneficial  impacts 

Detrimental  impacts 

Beneficial  impacts 

Detrimental  impacts 

Program 

(Report)  (Effect  size)  (p-value) 

(Report)  (Effect  size)  (p-value) 

(Report)  (Effect  size)  (p-value) 

(Report)  (Effect  size)  (p-value) 

SS 

Year  1 

Feelings  of  Safety 
(TRCS)  (.37)  (.216) 

Year  2 

Positive  Social  Behavior 
(PCR)  (-.14)  (.050) 

Feelings  of  Safety 
(TRCS)  (.39)  (.197) 

Year  3 

Feelings  of  Safety 
(TRCS)  (.52)  (.062) 

1 Out  of  the  140  comparisons  made  for  each  year,  7 would  be  expected  to  be  statistically  significant  at  the  .05  level  by  chance  (for  a total  of  21). 

2 Defined  as  impacts  that  were  not  statistically  significant  but  were  .25  standard  deviation  units  (absolute  value)  or  more  in  magnitude. 

NOTE:  Abbreviations  are 


ABC:  Academic  and  Behavioral  Competencies  Program 
CSP:  Competence  Support  Program 
LBW:  Love  In  a Big  World 
PA:  Positive  Action 

PATHS:  Promoting  Alternative  Thinking  Strategies 

4Rs:  The  4Rs  Program  (Reading,  Writing,  Respect,  and  Resolution) 

SS:  Second  Step 

CR:  Child  Report 

PCR:  Primary  Caregiver  Report 

TRS:  Teacher  Report  on  Student 

TRCS:  Teacher  Report  on  Classroom  and  School 

Blank  cell:  Finding  of  no  impact 

All  impact  estimates  were  calculated  using  regression  models  in  which  each  school  within  a program  was  weighted  equally.  The  standard  errors  of  all  estimates  account  for  design 
effects  due  to  unequal  weighting  and  the  clustering  of  students  within  schools.  Significance  is  based  on  p < .05.  The  number  of  results  found  significant  was  no  more  than  expected  by 
chance. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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The  lack  of  statistically  significant  impacts  found  in  the  combined-program  analysis  was  not  due  to  offsetting 
impacts  among  subgroups  defined  by  gender,  stayer  versus  new  entrant  status,  and  different  levels  of  initial 
student  risk.  A greater  number  of  significant  differences  were  found  than  would  be  expected  by  chance  in  the 
gender  and  initial  student  risk  analyses  (but  not  in  the  stayer  analysis);  however,  the  differences  did  not  favor 
any  one  subgroup,  and  only  in  seven  cases  were  they  replicated  across  the  years  (these  data  are  not  shown  in  a 
table).  The  analysis  by  gender  found  8 statistically  significant  differences  between  the  genders  out  of  54 
possible,  where  3 would  be  expected  by  chance;  half  of  the  significant  impacts  were  beneficial  impacts  and 
half  were  detrimental  impacts  (one  of  the  detrimental  impacts  was  replicated  in  Years  2 and  3).  The  analysis 
of  stayers  versus  new  entrants  found  no  statistically  significant  differences  in  the  36  comparisons.  For  the  five 
types  of  different  initial  risk,  the  analysis  found  41  statistically  significant  differences  among  three  levels  of 
student  risk  (low,  average,  and  high)  out  of  the  270  possible  (13  or  14  would  have  been  expected  to  be 
significant  by  chance);  of  these,  26  showed  more  beneficial  impacts  for  higher  risk  students  versus  lower  risk 
students  (4  of  these  were  replications  across  years),  and  15  showed  more  detrimental  impacts  for  higher  risk 
students  (2  of  these  were  replications  across  years).25 

The  analysis  of  the  fidelity  data  found  little  evidence  of  a relationship  between  high  fidelity  and  more 
beneficial  outcomes  (these  data  are  not  shown  in  a table).  The  number  of  significant  associations  found 
between  fidelity  and  beneficial  outcomes  was  higher  than  expected  by  chance  (5  associations  found  compared 
to  3 that  might  be  expected  by  chance  out  of  54  estimated  impacts)  but  4 of  the  5 significant  results  were  due 
to  detrimental  associations  between  low  fidelity  and  outcomes  (rather  than  beneficial  associations  between 
high  fidelity  and  outcomes). 

In  conclusion,  the  analysis  of  the  year-by-year  impacts  did  not  yield  evidence  that  the  seven  SACD  programs, 
combined  and  individually,  improved  student  social  and  character  development.  A small  number  of  findings 
were  statistically  significant  (but  no  more  than  would  be  expected  by  chance,  except  for  several  of  the 
subgroups)  or  substantively  important.  These  results  were  split  into  similar  numbers  of  beneficial  and 
detrimental  impacts;  that  is,  the  SACD  programs  improved  some  outcomes  but  worsened  others.  In  the 
majority  of  cases,  the  results  (both  beneficial  and  detrimental)  occurred  in  only  1 year  and  were  not  replicated 
across  the  3 years  of  the  study. 

Impacts  on  Growth  of  Student  Outcomes 

A growth  curve  analysis  was  done  to  examine  the  change  over  time  in  the  impacts  on  the  outcomes  between 
fall  2004  and  spring  2007. 26  The  growth  curve  analysis  used  the  same  covariates  and  compared  results  across 
the  same  subgroups  as  the  cross-sectional  analyses  to  ensure  the  comparability  of  the  results.  However,  it 
differed  from  the  cross-sectional  analyses  by  examining  the  estimated  impacts  on  the  trajectories  of  student 
outcomes  over  time,  rather  than  at  a point  in  time.  The  sample  of  students  for  the  growth  curve  analysis 
included  all  students  who  were  enrolled  in  one  of  the  study  schools  during  the  study  period  and  who 
completed  a survey  during  the  initial  data  collection  or  at  any  of  the  four  follow-up  survey  points.  The 
percentages  of  the  sample  with  responses  were  similar  for  treatment  and  control  schools  in  most  survey 
waves;  however,  the  percentages  were  statistically  significantly  higher  for  the  treatment  group  on  the  Child 
Report  and  Primary  Caregiver  Report  in  fall  2004  and  on  the  Child  Report  in  spring  2005.  There  was 
considerable  turnover  within  the  sample  by  spring  2007.  Across  the  three  survey  instruments  that  reported  on 
students,  about  two-thirds  of  the  sample  had  taken  part  in  the  original  fall  2004  survey  (66%  for  the  Child 
Report  and  the  Primary  Caregiver  Report,  and  68%  for  the  Teacher  Report  on  Student).  There  were  no 
statistically  significant  differences  in  the  level  of  turnover  by  treatment  status.27  The  18  child-level  outcome 
measures  collected  from  the  Child  Report,  Primary  Caregiver  Report,  and  Teacher  Report  on  Student  were 
used  in  the  growth  curve  analysis.  The  2 other  outcomes,  Feelings  of  Safety  and  Student  Support  for  Teacher, 


25  The  section  Year-By-Year  Subgroup  Analysis  of  Impacts  on  Students  in  chapter  1 provides  the  details  on  these  results. 

26  See  Impacts  on  Growth  of  Student  Outcomes  in  chapter  1 for  more  detail  on  the  growth  curve  analysis  and  results. 

27  For  details  see  table  1.36  in  chapter  1. 
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collected  by  the  Teacher  Report  on  Classroom  and  School  were  not  used  because  they  were  measured  at  the 
teacher  level. 

The  impacts  over  time  were  estimated  using  growth  curve  models  (with  time  since  implementation  of  the 
program  as  the  time  metric)  by  examining  treatment  and  control  group  differences  in  the  trajectories  of 
student  outcomes  during  the  study  while  accounting  for  clustering  at  the  school  level.  The  growth  curve 
models  were  estimated  using  a three-level  hierarchical  linear  model,  where  Level  1 corresponded  to  time  since 
implementation  of  the  program,  Level  2 to  students,  and  Level  3 to  schools.  The  models  included  the  same 
set  of  covariates  that  were  used  for  the  cross-sectional  analysis  except  they  excluded  the  initial  outcome 
measure  as  a covariate  because  it  was  used  as  the  outcome  measure  for  the  growth  curve  analysis  at  time  1 
(fall  2004).  Similarly,  sample  weights  were  used  in  all  analyses  to  (1)  give  each  program  equal  weight  within 
each  time  period;  (2)  give  each  school  equal  weight  in  each  program  (within  each  time  period);  and  (3)  give 
each  time  period  equal  weight  in  the  analysis.  However,  the  weights  were  not  adjusted  for  consent  and 
response  differences  across  classrooms  or  schools  because  the  population  of  students  within  the  schools 
changed  over  time  as  students  entered  and  left  the  schools.  Similar  to  the  year -by-year  analyses,  the  growth 
model  was  estimated  for  the  seven  programs  combined  and  for  each  program  individually.  In  addition,  the 
effects  of  the  combined  programs  by  subgroup  were  estimated  by  including  interaction  terms  between 
treatment  status,  time  since  implementation,  and  indicators  of  membership  in  subgroups. 

Growth  curve  effect  sizes  were  calculated  by  dividing  the  estimated  impact  of  the  treatment  on  the  outcome 
growth  trajectory  by  the  standard  deviation  of  that  outcome.28  The  growth  curve  estimates  the  change  in 
outcomes  over  1 year,  so  the  estimated  impact  of  the  treatment  on  the  growth  trajectory  equals  the  difference 
between  the  treatment  group’s  outcome  and  the  control  group’s  outcome,  on  average,  after  1 year  of  the 
study.29  The  effect  size  measures  the  number  of  standard  deviations  the  treatment  group  differs  from  the 
control  group  after  1 average  year  of  the  study,  making  it  analogous  to  the  effect  size  calculations  for  the 
cross-sectional  analysis. 

The  growth  curve  analysis  found  no  significant  effects  of  the  seven  SACD  programs  when  combined  (table 
H).  None  of  the  18  estimated  impacts  on  the  trajectories  of  child  outcomes  from  the  average  of  the  seven 
SACD  programs  were  statistically  significant.  The  estimated  effect  sizes  all  fell  below  .07  (absolute  value). 

The  lack  of  significant  effects  reflected  in  the  analysis  of  the  seven  programs  together  was  not  found  to  be 
due  to  differences  among  the  individual  programs.  The  results  from  the  analysis  of  individual  programs 
indicate  that  the  lack  of  significant  impacts  in  the  overall  evaluation  reflected  the  lack  of  significant  impacts  at 
the  program  level.  Six  statistically  significant  impacts  were  found  in  the  program  analyses,  the  same  number 
expected  by  chance  given  that  126  impacts  were  estimated.  Two  were  beneficial  impacts  and  four  were 
detrimental  impacts  (table  I). 


28  The  standard  deviation  was  calculated  using  data  for  the  weighted  control  group.  It  was  calculated  at  the  time  of  data 
collection  for  which  the  effect  size  (impact)  was  estimated. 

29  This  impact  estimate  takes  into  account  differences  between  the  initial  levels  of  the  outcome  for  the  treatment  and 
control  groups,  differences  in  their  covariates,  and  the  effects  of  clustering  at  the  school  level. 
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Table  H.  Impacts  on  growth  of  child  outcomes  from  combined-program  analysis 


Average  growth  in  the  score  per  year 

1 

Scale-Report 

Mean  score  at 
implementation2 

Treatment 

group 

Control 

group 

Impact  on 
growth3 

Effect 

4 

size 

Standard 
error  of 
impact 

p-value  of 
impact 

Social  and  Emotional  Competence  Domain 

Self-Efficacy  for  Peer  Interactions-CR  (+) 

2.95 

0.13 

0.13 

0.00 

0.00 

0.01 

0.942 

Normative  Beliefs  About  Aggression-CR  (-) 

1.23 

0.10 

0.07 

0.02 

0.04 

0.01 

0.115 

Empathy-CR  (+) 

2.41 

-0.14A 

-0.12 

-0.02 

-0.05 

0.01 

0.070 

Behavior  Domain 

Altruistic  Behavior-CR  (+) 

1.41 

-0.17 

-0.16 

-0.01 

-0.01 

0.02 

0.681 

Altruistic  Behavior-PCR  (+) 

2.32 

-0.03 

-0.03 

0.01 

0.01 

0.01 

0.616 

Altruistic  Behavior-TRS  (+) 

1.40 

-0.04 

0.00 

-0.04 

-0.07 

0.03 

0.224 

Positive  Social  Behavior-PCR  (+) 

2.99 

0.04 

0.03 

0.01 

0.02 

0.01 

0.217 

Positive  Social  Behavior-TRS  (+) 

3.00 

0.00 

0.01 

-0.01 

-0.01 

0.03 

0.671 

Problem  Behavior-CR  (-) 

0.24 

0.10 

0.08 

0.01 

0.02 

0.01 

0.351 

Problem  Behavior-PCR  (-) 

1.58 

-0.01 

-0.01 

0.00 

0.00 

0.01 

0.843 

Problem  Behavior-TRS  (-) 

1.38 

0.05 

0.05 

0.00 

0.00 

0.01 

0.909 

ADHD-Related  Behavior-TRS  (-) 

1.75 

-0.02 

-0.01 

-0.01 

-0.01 

0.02 

0.707 

Academics  Domain 

Engagement  with  Learning-CR  (+) 

3.69 

-0.03 

-0.03 

0.00 

-0.01 

0.01 

0.707 

Academic  Competence  and  Motivation-TRS  (+) 

2.87 

0.02 

0.03 

-0.01 

-0.01 

0.02 

0.590 

See  notes  at  end  of  table. 
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Table  H.  Impacts  on  growth  of  child  outcomes  from  combined-program  analysis — Continued 


Average  growth  in  the  score  per  year1 

Standard 


Scale-Report 

Mean  score  at 
implementation2 

Treatment 

group 

Control 

group 

Impact  on 
growth3 

Effect 

4 

size 

error  of 
impact 

p-value  of 
impact 

Perceptions  of  School  Climate  Domain 
Positive  School  Orientation-CR  (+) 

3.09 

-0.24 

-0.21 

-0.03 

-0.03 

0.02 

0.163 

Negative  School  Orientation-CR  (-) 

1.84 

0.10 

0.09 

0.01 

0.01 

0.02 

0.696 

Student  Afraid  at  School-CR  (-) 

2.38 

-0.08 

-0.08 

0.00 

0.00 

0.02 

0.956 

Victimization  at  School-CR  (-) 

0.76 

-0.03 

-0.02 

-0.01 

-0.01 

0.02 

0.368 

A Treatment  group  significantly  different  from  control  group  at  the  .10  to  > .05  level. 

1 Pertains  to  the  estimated  slope  of  the  outcome  for  the  treatment  or  control  groups. 

2 The  average  score  at  implementation  is  calculated  across  treatment  and  control  groups,  using  regression  models  for  adjustment  on  covariates. 

3 Estimated  difference  between  the  slope  of  the  treatment  and  control  groups. 

4 The  slope  of  the  treatment  group  minus  the  slope  of  the  control  group  divided  by  the  standard  deviation  of  the  outcome  for  the  combined-program  control  group  (the  standard 
deviation  is  calculated  without  accounting  for  school-level  clustering  or  regression  adjustments). 

NOTE:  Abbreviations  are 
CR:  Child  Report 
PCR:  Primary  Caregiver  Report 
TRS:  Teacher  Report  on  Student 
ADHD:  Attention  deficit  hyperactivity  disorder 

The  +/-  signs  in  parentheses  indicate  the  direction  of  a beneficial  outcome.  No  findings  were  found  statistically  significant  at  or  below  the  .05  level.  All  impact  estimates  were  calculated 
using  HLM  6.06. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Table  I.  Significant  impacts  from  the  growth  curve  analyses  of  the  individual  programs 


Program 

Significant  beneficial  impacts 
(Report)  (Effect  size)  (p-value) 

Significant  detrimental  impacts 
(Report)  (Effect  size)  (p-value) 

Total  significant*1 
impacts 

Total 

2 

4 

6 

ABC 

0 

CSP 

Victimization  at  School 
(CR)  (-.09)  (.050) 

1 

LBW 

Positive  School  Orientation 
(CR)  (-.13)  (.016) 

1 

PA 

0 

PATHS 

Academic  Competence 
(TRS)  (.08)  (.048) 

1 

4Fts 

0 

SS 

Engagement  with  Learning 
(CR)  (-.09)  (.021) 
Positive  Social  Behavior 
(TRS)  (-.19)  (.019) 
Empathy 

(CR)  (-0.13)  (.028) 

3 

* Significantly  different  from  zero  at  the  .05  level. 

1 Out  of  the  126  comparisons  made  (7  programs  times  18  outcomes),  6 would  be  expected  to  be  statistically  significant  at  the  .05 
level  by  chance. 

NOTE:  Abbreviations  are 

ABC:  Academic  and  Behavioral  Competencies  Program 
CSP:  Competence  Support  Program 
LBW:  Love  In  a Big  World 
PA:  Positive  Action 

PATHS:  Promoting  Alternative  Thinking  Strategies 

4Rs:  The  4Rs  Program  (Reading,  Writing,  Respect,  and  Resolution) 

SS:  Second  Step 
CR:  Child  Report 
TRS:  Teacher  Report  on  Student 
Blank  cell:  Finding  of  no  impact 

The  number  of  results  found  significant  was  no  more  than  expected  by  chance. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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The  lack  of  findings  from  the  combined-program  analysis  was  not  found  to  be  due  to  differences  among 
subgroups  (these  data  are  not  shown  in  a table).  The  subgroup  analyses  did  not  find  evidence  regarding 
differential  impacts  of  the  seven  SACD  programs,  combined,  on  the  subgroups.  For  the  gender  subgroup 
analysis,  none  of  the  36  estimated  impacts  were  statistically  significant.  For  the  analysis  of  new  entrants,  1 
impact  was  found  to  differ  significantly  and  detrimentally  from  the  impact  on  members  of  the  original 
sample,  which  was  no  more  than  would  be  expected  by  chance.  There  were  6 outcomes  out  of  90  estimated 
for  which  growth  trajectories  differed  significantly  across  initial  risk  levels  (4  to  5 would  have  been  expected 
by  chance);  for  4 of  the  6 outcomes,  the  programs  had  more  beneficial  impacts  for  children  with  higher  initial 
risk  levels,  while  for  2 outcomes  they  had  more  detrimental  impacts  for  children  with  higher  initial  risk  levels. 

Discussion 

The  year-by-year  analysis  and  the  growth  curve  analysis  did  not  find  that  the  seven  SACD  programs 
improved  student  outcomes  when  considered  together,  individually  by  program,  or  for  specific  subgroups. 
For  the  combined-program  analyses,  the  year-by-year  analysis  found  fewer  significant  impacts  than  expected 
by  chance  (2  out  of  60  estimated  impacts)  and  the  growth  curve  analysis  found  no  significant  impacts.  For  the 
individual  program  analyses,  the  year-by-year  analysis  found  fewer  significant  impacts  than  expected  by 
chance  (16  out  of  420  estimated  impacts),  with  9 having  beneficial  impacts  and  7 having  detrimental  impacts. 
The  growth  curve  analyses  of  the  individual  programs  found  the  same  number  of  impacts  as  expected  by 
chance  (6  out  of  126  estimated  impacts),  with  2 having  beneficial  impacts  and  4 having  detrimental  impacts. 
For  the  subgroup  analyses,  the  year-by-year  analyses  found  more  significant  impacts  than  expected  by  chance 
for  gender  (8  out  of  54  estimated  impacts)  and  initial  risk  levels  (41  out  of  270  estimated  impacts).  For 
gender,  half  of  the  significant  impacts  showed  a beneficial  impact  of  the  intervention  and  half  showed  a 
detrimental  impact.  For  initial  risk  levels,  26  showed  a beneficial  impact  of  the  intervention  on  high-risk 
students  and  15  showed  a detrimental  impact  on  high-risk  students.  The  growth  analysis  found  fewer 
significant  impacts  than  expected  by  chance  for  all  the  subgroups  except  those  based  on  initial  risk  levels  (6 
out  of  90  estimated  impacts).  Four  of  the  6 significant  impacts  were  beneficial  and  2 were  detrimental.  In 
sum,  the  SACD  combined-program  evaluation  provides  no  evidence  that  the  seven  universal,  schoolwide 
programs  improved  students’  social  and  character  development. 

Several  explanations  for  this  finding  can  be  considered:  (1)  failure  of  the  conceptualization  and  design  of  the 
intervention,  (2)  weak  implementation  of  the  intervention,  (3)  nonsubstantial  differences  in  the  level  of 
SACD  activities  in  the  treatment  and  control  schools,  and  (4)  methodological  limitations  of  the  evaluation. 

Failure  of  the  conceptualization  and  design  of  the  intervention  refers  to  the  possibility  that  the  seven 
programs  tested  might  not  have  altered  students’  social  and  character  development  in  the  expected  ways 
because  the  theories  underlying  them  or  the  combinations  of  activities  chosen  to  bring  about  the  desired 
changes  in  students’  attitudes  and  behaviors  were  inadequate  for  the  purpose.  For  example,  one  alternative 
view  to  that  adopted  in  the  SACD  evaluation  is  that  only  a subset  of  elementary-aged  children  has  deficits  in 
social  behavior  and  character,  and  these  deficits  require  a more  targeted,  more  intensive  intervention  than 
schoolwide  programs  can  provide.  Therefore,  for  a school-based  program  to  be  effective,  a combination  of 
schoolwide  and  targeted  activities  might  be  required  for  the  intervention  to  make  a significant  difference  in 
student  outcomes  (e.g.,  see  Conduct  Problems  Prevention  Research  Group  1999).  Such  explanations,  if 
confirmed  by  other  studies,  would  lead  toward  focusing  more  effort  on  understanding  how  social  and 
character  development  occurs  among  elementary-aged  children,  how  this  development  can  be  affected,  and 
what  types  of  practices  in  classrooms  and  schools  can  be  used  to  bring  about  desired  effects. 

Weak  implementation  of  the  intervention  refers  to  the  possibility  that,  although  the  intervention  might  have 
been  well  conceived  and  well  designed,  the  treatment  schools  did  not  implement  the  practices  effectively  on 
average.  Weakly  implemented  programs  may  not  have  positive  impacts  on  students.  The  SACD  evaluation 
examined  seven  different  SACD  programs,  each  with  unique  features,  and  the  fidelity  rating  that  was  used 
measured  fidelity  of  implementation  relative  to  the  targets  established  for  each  specific  intervention.  In  this 
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way,  the  ratings  were  standardized  relative  to  each  site’s  program-specific  benchmarks,  allowing  them  to  be 
compared  across  programs  and  years.  The  analysis  of  the  fidelity  data  found  little  evidence  of  a relationship 
between  high  fidelity  and  more  beneficial  outcomes.  The  number  of  significant  associations  found  between 
fidelity  and  beneficial  outcomes  was  higher  than  expected  by  chance  (5  found,  with  3 expected  by  chance  out 
of  54  estimated  impacts),  but  4 of  the  5 significant  results  were  due  to  detrimental  associations  between  low 
fidelity  and  outcomes  rather  than  beneficial  associations  between  high  fidelity  and  outcomes.  The  approach 
used  to  obtain  comparable  ratings  required  two  compromises.  First,  it  could  not  account  for  differences 
among  the  programs’  implementation  standards — for  example,  whether  programs  differed  in  how  difficult 
they  were  to  implement.  Second,  it  provided  little  information  about  why  implementation  was  of  a certain 
quality.  As  a result,  the  SACD  evaluation  fidelity  measure  may  not  provide  adequate  information  about 
whether  low  fidelity  might  have  been  the  reason  behind  the  lack  of  significant  findings.  The  fidelity  measures 
used  by  each  research  team,  which  used  team-chosen  criteria,  may  provide  additional  information  on  how 
well  each  program  was  implemented  in  each  treatment  school. 

Nonsubstantial  differences  in  the  level  of  SACD  activities  refers  to  the  possibility  that  the  implementation 
differences  between  the  treatment  and  control  schools  were  not  great  enough  to  generate  statistically 
significant  differences  in  student  outcomes.  Like  the  treatment  schools,  the  control  schools  joined  the  study 
with  a willingness  to  implement  a SACD  program  showing  a willingness  to  promote  social  and  character 
development.  In  addition,  some  of  the  sites  were  located  in  states  where  legislation  required  or  promoted 
such  activities.  The  control  group  represented  “standard  practice,”  which  included  the  reported  use  of  SACD 
activities  in  the  classroom.  For  example,  86  percent  to  90  percent  of  control  teachers  reported  using  activities 
to  promote  any  one  of  the  six  SACD  goals.  While  a statistically  significant  larger  percentage  of  teachers  in  the 
treatment  schools  (95%  to  96%)  reported  conducting  such  activities,  the  5-  to  10-percentage-point 
differences  may  not  have  been  large  enough  to  lead  to  improved  student  outcomes.  At  the  same  time,  the 
significant  differences  between  treatment  and  control  teacher  reports  were  larger  than  5 to  10  percentage 
points  for  other  responses  regarding  the  use  of  SACD  activities.  For  example,  the  differences  between 
treatment  and  control  teachers  regarding  the  use  of  activities  from  specifically  named  programs  were  29  to  34 
percentage  points  across  the  3 years.  These  results,  plus  the  finding  that  treatment  teachers  reported  greater 
use  of  some  instructional  materials  and  methods  to  promote  SACD  goals,  provide  evidence  that  the 
treatment  teachers  were  making  a more  intensive  effort  to  promote  social  and  character  development. 

There  are  three  methodological  limitations  of  the  evaluation  that  may  have  contributed  to  the  finding  of  no 
impacts  on  student  outcomes. 

First,  the  evaluation  relied  on  self-reported  data  by  teachers  and  principals  regarding  the  use  of  SACD 
activities.  Observational  studies  were  not  done  to  validate  these  reports.  If  treatment  teachers  over-reported 
their  use  of  SACD  activities  (possibly  because  they  felt  an  expectation  to  report  high  use  given  that  a SACD 
program  was  being  implemented  in  their  school),  the  impacts  of  the  treatment  could  be  misestimated.  That  is, 
if  there  really  were  no  differences  in  the  levels  of  SACD  activities  between  the  treatment  and  control  groups, 
then  a lack  of  effects  might  be  expected.  However,  treatment  teachers  did  differ  in  their  reported  use  of  any 
activities  to  promote  one  of  the  six  SACD  goals  (95%  to  96%  reported  such  use)  versus  their  reported  use  of 
any  activities  from  named  programs  (68%  to  72%)  This,  while  not  ruling  out  the  possibility  of  systematic 
over-reporting,  might  suggest  that  some  teachers  were  candid  in  their  reporting  on  their  use  of  the  treatment 
programs.  The  research  teams  used  observations  or  product  measures  to  check  implementation  of  their 
specific  programs,  and  the  results  from  this  work  may  provide  additional  evidence  about  the  potential 
importance  of  over-reporting  of  implementation  by  treatment  teachers. 

A second  methodological  limitation  was  that  student-provided  data  (used  for  12  of  the  20  outcomes)  were 
not  available  for  36  percent  to  39  percent  of  students,  depending  on  the  year,  because  primary  caregivers  did 
not  provide  written  consent  or  students  did  not  assent  to  take  part  in  the  study  (primary  caregiver  data,  used 
for  three  outcomes,  were  not  available  for  49%  to  54%  of  students).  It  is  possible  that  students  included  in 
the  study  differed  from  those  not  included  due  to  an  absence  of  data.  As  the  study  did  not  collect  descriptive 
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data  on  the  nonobserved  students,  the  existence  of  such  differences  could  not  be  determined,  and  how  the 
inclusion  of  these  students  in  the  study  would  have  affected  the  findings  is  unknown.  Given  the  few 
statistically  significant  and  substantively  important  impacts  found  with  the  existing  sample,  there  would  need 
to  be  a large  and  consistent  impact  on  the  nonobserved  students  (had  they  been  included)  to  change  the 
findings.  For  example,  because  two-thirds  of  the  population  who  were  observed  received  an  average  impact 
of  zero,  the  nonobserved  one-third  would  need  to  have  received  an  average  impact  of  nearly  one-third  of  a 
standard  deviation  to  bring  the  overall  mean  to  one -tenth  of  a standard  deviation  unit.  In  addition,  because 
the  subgroup  analyses  did  not  find  systematically  significant  impacts,  there  is  no  evidence  that  should  these 
missing  students  come  from  one  of  the  subgroups  (e.g.,  higher  initial  risk)  their  inclusion  would  change  the 
findings. 

A third  methodological  limitation  was  the  sample  size  for  the  individual  program  evaluations.  The  combined 
analysis  of  all  seven  programs  provided  a sample  size  sufficient  to  detect  student -level  impacts  at  minimum 
detectable  effect  sizes  (MDES)  ranging  from  0.03  to  0.23  standard  deviations  (with  more  than  75%  of  them 
below  0.10  standard  deviations).  The  power  to  detect  impacts  at  the  level  of  the  individual  programs  was 
more  limited  (the  MDES  ranged  from  0.09  to  1.04  over  the  3 years),  and  individual  program-level  effects 
might  have  been  missed.  To  address  this  limitation  at  the  program  level,  nonsignificant  impacts  of  at  least 
0.25  standard  deviations  were  identified  as  substantively  important  results.  Only  a small  number  of  these  were 
identified,  with  an  almost  equal  division  into  beneficial  and  detrimental  effects. 

The  seven  programs  tested  in  the  SACD  evaluation  were  similar  in  being  coherent,  universal,  school-based 
programs  to  promote  social  and  character  development  of  third-  to  fifth-grade  students.  They  were  diverse 
both  in  their  specific  goals  and  in  their  approaches  to  promoting  social  and  character  development  for  all 
students.  In  addition,  they  were  evaluated  in  diverse  types  of  locales  in  which  schools  served  communities 
with  very  different  ethnic  and  socioeconomic  compositions.  On  average,  the  seven  programs  did  not  improve 
students’  social  and  emotional  competence,  behavior,  academic  achievement,  and  student  and  teacher 
perceptions  of  school  climate.  In  addition,  although  the  numbers  of  schools  and  students  in  each  program 
were  not  always  sufficient  to  support  firm  conclusions  at  the  program  level,  the  patterns  of  estimated  impacts 
for  each  program  were  largely  similar:  students’  outcomes  were  not  affected. 
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The  Social  and  Character  Development  (SACD)  Research  Program  was  created  in  response  to  the  need  for 
rigorous  evaluations  of  universal  school-based  programs  designed  to  help  schools  foster  positive  behaviors 
among  students  (e.g.,  behaviors  illustrating  good  character  and  social-emotional  competence),  reduce  negative 
behaviors  (e.g.,  aggression  and  conduct  problems),  and  ultimately  improve  students’  academic  performance. 
For  this  evaluation,  such  programs  were  termed  SACD  programs,  the  goals  they  intended  to  promote  were 
termed  SACD  goals,  and  the  activities  they  employed  to  promote  those  goals  were  termed  SACD  activities.1 

The  SACD  Research  Program  built  on  advances  in  understanding  the  development  of  social  competence  and 
problem  behaviors  occurring  in  the  fields  of  developmental,  social,  community,  behavioral,  and  cognitive 
psychology  (e.g.,  Damon  1999;  Eisenberg  2000;  Larson  2000;  Seligman  and  Csikszentmihalyi  2000),  and  from 
intervention  and  evaluation  research  in  the  areas  of  prevention  science,  public  health,  youth  development, 
and  character  education  (e.g.,  Catalano  et  al.  2004;  Dahlberg  and  Simon  2006;  Emler  1996;  Flay  2002).  Both 
federal  legislation  (such  as  the  No  Child  Left  Behind  Act  of  2001)  and  state  legislation  (Limber  and  Small 
2003)  have  supported  prevention  and  intervention  programs,  and  the  SACD  Research  Program  sought  to  add 
to  the  research  base  underlying  such  efforts. 

In  2003,  the  Institute  of  Education  Sciences  (IES)  in  collaboration  with  the  Division  of  Violence  Prevention 
(DVP)  in  the  National  Center  for  Injury  Prevention  and  Control,  Centers  for  Disease  Control  and  Prevention 
(CDC)  began  the  SACD  Research  Program  to  evaluate  multiple  school-based  programs  in  a consistent 
manner.  Under  a competitive  application  process,  applicants  proposed  programs  to  be  evaluated  as  long  as 
the  programs  (a)  had  either  preliminary  evidence  of  success  or  a history  of  previous  implementation  in 
schools,  (b)  aimed  to  influence  social  development  and  behavior  outcomes,  and  (c)  utilized  a universal 
approach  to  be  implemented  in  all  elementary  school  classrooms.  Applications  submitted  to  IES  were  peer 
reviewed,  and  seven  research  institutions  were  funded  under  cooperative  agreements  for  a 3-year  evaluation 
of  seven  universal  school-based  SACD  programs.  At  each  of  seven  sites,  one  research  team  recruited  and 
randomized  10  to  18  schools  to  either  continue  their  current  practice  or  implement  a coherent  program 
targeting  social  and  behavioral  outcomes.  The  programs  employed  activities  to  promote  six  SACD  goals 
(character  education,  violence  prevention  and  peace  promotion,  social  and  emotional  development,  tolerance 
and  diversity,  risk  prevention  and  health  promotion,  and  civic  responsibility  and  community  service)  as  well 
as  behavior  management.  The  programs  were  coherent  in  that  their  activities  were  integrated  and  logically 
organized  based  on  a theory  of  action  (that  differed  among  the  programs),  school-based  in  that  they  were 
implemented  in  the  schools  by  school  personnel,  and  universal  in  that  they  were  to  be  implemented  for  all 
students  in  all  elementary  classrooms.2  The  research  institutions  participating  in  the  study  and  the  programs 
evaluated  included  the  following: 


1 The  SACD  evaluation  examined  seven  SACD  programs  that  employed  SACD  activities  to  promote  behavior 
management  and  six  SACD  goals:  (1)  violence  prevention  and  peace  promotion,  (2)  social  and  emotional  development, 
(3)  character  education,  (4)  tolerance  and  diversity,  (5)  risk  prevention  and  health  promotion,  and  (6)  civic  responsibility 
and  community  service. 

2 As  used  here,  “elementary  classrooms”  refers  to  grades  K-5.  During  the  study,  there  were  five  exceptions  to 
implementation  in  all  elementary  classrooms  in  the  treatment  schools.  In  Year  1 (the  2004-05  school  year),  all  treatment 
schools  implemented  the  intervention  in  their  K-5  classrooms.  In  Year  2 (2005-06),  one  school  implemented  the  full 
intervention  in  fourth  grade  only  (the  cohort  grade  that  year),  and  one  school  could  not  implement  in  fifth  grade  because 
that  grade  had  been  transferred  to  a middle  school.  In  Year  3 (2006-07),  one  school  implemented  the  intervention  in 
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Research  Institution 

University  at  Buffalo,  The  State  University  of 
New  York 

University  of  North  Carolina  at  Chapel  Hill 

Vanderbilt  University 

Oregon  State  University 

The  Children’s  Institute 

New  York  University 

University  of  Maryland 


Program  Evaluated 

Academic  and  behavioral  Competencies  Program 
(ABC) 

Competence  Support  Program  (CSP) 

Cove  In  a Big  World  (LBW) 

Positive  Action  (PA) 

Promoting  Alternative  Thinking  Strategies  (PATHS) 

The  4Rs  Program  (Reading,  Writing  Respect  & 
Resolution)  (4Rs) 

Second  Step  (SS) 


In  addition  to  conducting  their  own  program  evaluations,  applicants  were  expected  to  participate  in  the 
collection  of  data,  facilitated  by  an  independent  evaluation  contractor.  Under  another  competitive  peer-review 
process,  Mathematica  Policy  Research,  Inc.  (MPR)  was  selected  to  independently  evaluate  the  seven 
programs. 

The  evaluation  of  the  SACD  programs  thus  occurred  at  both  the  program-specific  and  multiprogram  level.  In 
the  program-specific  studies,  each  research  team  collected  data  to  answer  questions  that  were  specific  to  the 
program  under  evaluation.  For  example,  in  the  program-specific  studies,  investigators  sought  to  answer 
questions  about  the  effects  of  the  program  on  targeted  proximal  outcomes  and  about  the  importance  of 
program  implementation  quality.  Research  teams  designed  their  own  studies  and  were  expected  to  report 
their  findings  in  the  relevant  literature. 

For  the  multiprogram  evaluation,  an  independent  team  from  MPR  assessed  the  effect  of  the  seven  SACD 
programs  through  an  evaluation  of  all  programs  together  using  (1)  standardized  data  collection  for  all  sites; 
(2)  a common  set  of  descriptive  measures  on  the  types  and  level  of  SACD  activities  taking  place  at  both  the 
treatment  and  control  schools;  (3)  a common  set  of  outcome  measures  grouped  under  four  outcome 
domains:  Social  and  Emotional  Competence,  Behavior,  Academics,  and  Perceptions  of  School  Climate;  and 
(4)  a uniform  statistical  analysis.  The  effects  of  each  program  were  also  assessed  separately  to  identify  any 
contrasts  with  the  overall  findings  across  programs.  The  evaluation  followed  one  cohort  of  third-grade 
students  at  84  schools  for  3 years  through  fifth  grade.  Initial  data  were  collected  in  fall  2004,  and  follow-up 
data  were  collected  in  spring  2005,  fall  2005,  spring  2006,  and  spring  2007.  In  addition,  four  research  teams 
added  a second  cohort  of  third -graders  at  12  schools,  where  initial  data  were  collected  in  fall  2005  and  follow- 
up data  collected  in  spring  2006  and  spring  2007  (at  the  end  of  third  and  fourth  grades)  in  order  to  increase 
sample  sizes  and  the  associated  power  of  the  program-specific  analyses. 

This  report  contains  the  results  for  the  3-year  multiprogram  evaluation  and  focuses  on  the  first  cohort  of 
students.  It  details  the  impacts  of  the  seven  SACD  programs  after  1 , 2,  and  3 school  years  of  implementation 
in  the  treatment  schools  and  also  estimates  the  effect  of  the  programs  on  students’  growth  in  social  and 
character  development  over  the  3 years.  Chapter  1 discusses  the  evaluation  of  the  seven  programs  when 
considered  together  as  a subset  of  universal  school-based  programs  that  aim  to  affect  student  social 
development  and  behavior.  Chapter  1 also  provides  summary  results  for  each  program.  The  seven  chapters 
that  follow  detail  the  findings  for  each  individual  program.  There  are  two  appendixes:  appendix  A examines 
whether  the  addition  of  the  smaller  second  cohort  of  students  for  four  of  the  seven  programs  affected  the 
results  and  appendix  B contains  additional  technical  information  concerning  the  analyses. 


third  through  fifth  grades,  one  school  implemented  in  fifth  grade  only  (the  cohort  grade  that  year),  and  one  school’s  fifth 
grade  attended  a middle  school  so  it  was  not  able  to  implement  the  intervention  with  its  cohort  grade. 
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Social  Development  and  Risk  Prevention 

By  8 to  12  years  of  age,  children  typically  have  well-developed  skills  of  self-regulation,  perspective  taking, 
emotional  understanding,  and  caring  for  others.  Middle  childhood  is  also  a time  when  children’s  beliefs  about 
aggression  and  conflict  resolution  skills  are  developing  (e.g.,  Samples  and  Aber  1998).  Because  these  social 
competencies  underlie  many  primary  developmental  tasks  of  middle  childhood  (e.g.,  development  of  mutual 
friendships),  researchers  predict  that  deficits  in  these  social  skills  will  likely  lead  to  the  emergence  of  problem 
behaviors. 

Research  points  to  several  key  intrapersonal  factors  that  increase  a child’s  risk  for  behavior  problems, 
including  children’s  beliefs  about  the  acceptability  of  aggression  (Guerra  et  al.  1995),  lack  of  problem-solving 
skills  (Slaby  and  Guerra  1988),  attention  problems  and  hyperactivity  (Farrington  1989;  Rey,  Sawyer,  and  Prior 
2005),  and  attribution  of  aggression  to  others’  innocuous  behavior  (Dodge  and  Coie  1987).  Family,  peer,  and 
community  risk  factors  also  increase  the  likelihood  of  behavior  problems.  These  risk  factors  include  poor 
parental  supervision  and  monitoring  (Gorman-Smith  et  al.  1996),  association  with  aggressive  peers 
(Farrington  and  Flawkins  1991;  Flenry,  Tolan,  and  Gorman-Smith  2001),  community  disorganization 
(Sampson  2000),  and  exposure  to  community  violence  (Schwartz  and  Proctor  2000). 

Researchers  have  hypothesized  that  programs  targeting  the  risk  factors  described  above  will  be  effective  in 
reducing  problem  behaviors  among  children.  Over  time,  however,  there  has  been  increasing  recognition  that 
prevention  efforts  should  not  focus  on  risk  alone  but  take  an  integrative  approach  to  reducing  risk  and 
simultaneously  fostering  protective  factors  by  developing  social  competence  in  young  people  (Catalano  et  al. 
2004;  Weissberg,  Kumpfer,  and  Seligman  2003).  Many  protective  factors  that  are  hypothesized  to  promote 
social  competence  are  also  thought  to  inoculate  children  from  developing  externalizing  behavior  problems. 
Although  the  research  literature  on  protective  factors  that  relate  to  risk  has  been  slow  in  developing,  evidence 
suggests  that  socially  adaptive  interpersonal  problem-solving  (Wentzel  1991),  empathy  (Schultz,  Izard,  and 
Bear  2004),  and  parental  use  of  positive  rewards  and  encouragement  to  support  appropriate  behavior 
(Gorman-Smith  et  al.  2000)  are  related  to  a lower  occurrence  of  problem  behaviors,  such  as  aggression  and 
delinquency. 

From  an  interventionist  and  public  health  perspective,  schools  are  an  obvious  site  for  addressing  positive 
youth  development  and  prevention  efforts  because  of  universal  access  to  children  over  time  that,  in  turn, 
allows  for  efficient  distribution  of  these  efforts  to  a comprehensive  population  of  youth.  Elementary  school  is 
thought  to  be  a critical  time  for  prevention;  7 is  the  average  age  at  which  students  start  down  the  path  of 
problem  behavior  (Thornberry,  Huizinga,  and  Loeber  2004).  Most  types  of  antisocial  behavior  are  already 
evident  by  third  grade  (Loeber  and  Stouthamer-Loeber  1998),  and  problem  behaviors  in  childhood  are 
predictive  of  violence  and  other  antisocial  behavior  later  in  adolescence  and  adulthood  (Moffitt  1993). 

Programs  that  address  social  development  and  behavior  are  also  attractive  to  school  administrators  because 
of  the  prevalence  of  problem  behaviors  that  teachers  perceive  as  interfering  with  their  ability  to  teach  and 
students’  ability  to  learn  (Mansfield  et  al.  1991).  For  example,  in  the  2005-06  school  year,  21  percent  of 
primary  schools  reported  occurrences  of  student  bullying  at  least  once  per  week,  12  percent  of  schools 
reported  student  acts  of  disrespect  for  teachers  once  per  week,  and  28  percent  of  12-  to  18-year-old  students 
reported  that  they  had  been  bullied  at  school  during  the  6 months  prior  to  the  survey  (Dinkes,  Cataldi,  and 
Lin-Kelly  2007).  A substantial  body  of  literature  has  shown  that  disruptive  classroom  behavior,  conduct 
problems,  aggression,  delinquency,  and  substance  use  are  associated  with  poor  academic  achievement  and 
with  a lack  of  student  feelings  of  school  connectedness  and  involvement  (Bennett  et  al.  2003;  Farrington 
1989;  Fors,  Crepaz,  and  Hayes  1999;  McCord  et  al.  2000;  Najaka,  Gottfredson,  and  Wilson  2001;  O’Donnell, 
Hawkins,  and  Abbott  1995;  Trzesniewski  et  al.  2006).  On  the  positive  side,  social  competencies  have  been 
linked  with  higher  levels  of  achievement  and  school  adjustment  (Carlson  et  al.  1999;  Malecki  and  Elliot  2002; 
Wentzel  1993). 
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One  school  element  that  has  gained  increasing  attention  in  recent  years  is  the  creation  of  a positive  school 
climate.  Researchers  have  theorized  that  the  development  of  a warm,  caring  community  within  a school  might 
reduce  student  problem  behaviors,  such  as  aggression  and  bullying  (Battistich  et  al.  1997;  McEvoy  and 
Welker  2000;  Orpinas  and  Horne  2006);  however,  research  directly  testing  this  hypothesis  is  limited.  Overall, 
the  abundance  of  evidence  supporting  relations  among  social  competence,  problem  behavior,  and 
achievement,  and  the  development  of  theories  surrounding  the  potential  effects  of  a positive  school  climate 
on  student  outcomes,  has  motivated  the  development  and,  in  turn,  evaluation  of  school-based  programs. 

Universal  School-Based  Social  and  Character  Development  Programs 

Schools  commonly  use  support  services,  intervention  curricula,  and  discipline  management  strategies  to 
promote  social  and  character  development  and  prevent  problem  behavior  (Crosse  et  al.  2001;  Gottfredson 
and  Gottfredson  2001).  Many  of  these  programs  are  targeted  programs;  that  is,  they  focus  on  addressing 
behavior  problems  for  children  who  are  at  risk  or  for  children  who  are  already  exhibiting  adjustment 
difficulties  (see  Mytton  et  al.  2006  for  a review  of  the  effectiveness  of  such  approaches).  Alternatively,  a 
universal  approach  can  be  taken  to  influence  the  attitudes  and  behaviors  of  all  children  in  the  general  student 
population.  Universal  approaches  focus  on  preventing  problems  before  they  occur  by  addressing  factors  that 
place  youth  at  risk  for  problem  behavior  and  promoting  factors  that  foster  positive  youth  development 
(Walker  and  Shinn  2002). 

Universal  school-based  prevention  programs  have  followed  different  theoretical  traditions.  Character 
education  programs  teach  moral  values  through  the  curriculum  and  attempt  to  create  a climate  of  caring  and 
moral  discipline  (Lickona  1993).  Social  and  emotional  learning  programs  stress  goal-setting,  emotion 
identification,  responsible  decisionmaking,  perspective-taking,  and  effective  interpersonal  skills,  within  a 
caring  and  engaging  school  climate  (Greenberg  et  al.  2003;  Stage  and  Quiroz  1997).  Through  systematic 
assessment  and  evaluation  of  behavior,  Behavior  management  approaches  utilize  learning  theory  to  apply 
strategies  such  as  positive  reinforcement,  consistent  schoolwide  discipline,  and  antecedent  control  to 
minimize  disruptive  and  aggressive  behaviors  and  promote  prosocial  behaviors  in  all  settings  in  the  school 
(Kazdin  2001).  Primary  youth  violence  prevention  approaches  identify  the  individual,  relationship,  and 
environmental  factors  that  place  youth  at  risk  for  engaging  in  violence -related  behaviors.  They  implement 
strategies  that  modify  those  risk  factors,  such  as  by  changing  attitudes,  beliefs,  behaviors,  and  environments, 
to  disrupt  developmental  pathways  to  violence  (e.g.,  through  cognitive-behavioral  and  social  skill  training, 
changing  peer  group  norms,  and  modifying  school  settings  via  teacher  training  and  setting  school 
administrative  policies  and  rules;  Dahlberg  1998).  What  each  of  these  universal  school-based  approaches  has 
in  common  is  the  desired  goal  of  promoting  students’  social  development  and  reducing  engagement  in 
problem  behavior.  As  a group,  these  approaches  aim  to  improve  social  and  character  development. 

Evidence  of  the  effectiveness  of  universal  school-based  programs  to  promote  social  and  character 
development  is  mixed.  Some  recent  meta-analyses  suggest  that  universal  school-based  approaches  to 
preventing  aggressive  behavior  can  be  effective  (Hahn  et  al.  2007;  Wilson  and  Lipsey  2007).  Other  rigorous 
studies  and  meta-analyses  have  shown  significantly  positive  effects  on  some  outcomes  of  interest  but 
indeterminate  or  non-statistically  significant  effects  on  other  outcomes,  no  significant  effects  at  all,  and  even 
potentially  negative  effects  (e.g.,  CPPRG  1999;  Grossman  et  al.  1997;  Flannery  et  al.  2003,  Park-Higgerson  et 
al.  2008,  Merrell  et  al.  2008). 

In  addition,  some  universal,  school-based  SACD  programs  have  yet  to  be  evaluated,  while  others  have  been 
evaluated  using  nonrigorous  methodology,  design,  and  analysis  (e.g.,  small  sample  sizes  and  low  statistical 
power,  severe  attrition,  lack  of  randomization  to  condition,  and  inappropriate  level  of  analysis).  For  example, 
at  the  time  this  report  was  written,  the  What  Works  Clearinghouse  had  reviewed  55  evaluation  studies  of  13 
character  education  programs  and  found  that  two-thirds  of  these  studies  either  did  not  meet  Clearinghouse 
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standards  or  met  its  standards  with  reservations  (What  Works  Clearinghouse,  n.d.-a).3  The  What  Works 
Clearinghouse  also  identified  14  additional  character  education  programs  for  which  no  evaluations  were 
found. 

Given  the  methodological  limitations  of  many  previously  conducted  studies,  the  mixed  findings  from 
previous  rigorous  evaluations,  and  the  theoretical  rationales  for  conducting  school-based  programs, 
researchers  have  called  for  further  evaluation  of  prevention  programs  in  the  elementary  school  years  to  assist 
in  our  understanding  of  whether  middle  childhood  is  an  optimal  time  for  prevention  (Tolan,  Guerra,  and 
Kendall  1995).  In  particular,  there  has  been  a call  for  more  rigorous  evaluation  of  school -based  programs  in 
which  a universal  approach  to  development  and  prevention  is  used,  including  random  assignment  to 
condition;  documentation  of  control  group  practice;  reliable,  valid,  and  age-appropriate  measures  from 
multiple  data  sources;  short-  and  long-term  follow-up  of  students;  sufficient  statistical  power;  appropriate 
multilevel  analyses;  and  exploration  of  mediator  and  moderator  variables  (Farrell  and  Camou  2006; 
Greenberg  2004;  Weissberg,  Kumpfer,  and  Seligman  2003). 

One  difficulty'  with  interpreting  the  results  of  the  existing  evaluation  literature  has  been  the  use  of  different 
measurement  strategies  to  assess  similar  social  and  behavioral  outcomes.  That  is,  similar  outcomes  have  been 
assessed  using  different  measures  across  evaluations.  Meta-analyses  have  been  used  as  one  approach  to 
summarizing  the  effectiveness  of  universal  programs  on  behavior  by  clustering  outcomes  by  conceptual 
domains  (e.g.,  aggression,  disruptive  behavior;  Wilson  and  Lipsey  2007).  Another  approach,  one  taken  by  the 
SACD  Research  Program,  is  to  use  a common  set  of  measures  to  evaluate  multiple  programs  on  the  same 
outcomes  of  interest. 

Universal  SACD  programs  both  share  and  differ  in  their  structures,  approaches,  and  theories  of  change.  To 
assist  in  developing  hypotheses  and  selecting  measures  for  the  multiprogram  evaluation,  a global  conceptual 
model  was  developed  within  which  all  seven  programs  could  be  included,  although  none  would  strictly  follow 
the  entire  model.  The  model  was  used  to  help  in  the  design  of  the  multiprogram  evaluation  but  was  not  to  be 
tested  itself.  Figure  1.1  shows  the  set  of  common  structures  and  approaches  on  which  the  seven  SACD 
programs  draw.  It  identifies  a set  of  moderating  factors  expected  to  influence  the  success  of  all  programs; 
these  include  characteristics  of  the  student,  the  student’s  family,  the  school,  and  the  community,  as  well  as  the 
program.  The  model  shows  the  four  outcome  domains  (Social  and  Emotional  Competence,  Behavior, 
Academics,  and  Perceptions  of  School  Climate)  expected  to  be  affected  by  the  SACD  programs  and  the 
specific  outcomes  that  were  to  be  examined  within  each  domain.  The  pathways  in  the  model  show  that,  in 
general,  the  programs  were  expected  to  increase  the  proximal  domains  of  improving  social  competencies  and 
perceptions  of  school  climate  and,  directly  and  indirectly  through  the  proximal  domains,  to  affect  the  distal 
domains  of  improving  students’  behavior  and  academics. 

Research  Questions 

All  seven  of  the  programs  in  the  SACD  multiprogram  evaluation  were  universal,  school-based  interventions 
that  targeted  social  and  behavioral  outcomes.  The  evaluation  examined  both  the  average  effect  of  all  seven 
programs  combined  and  the  average  effect  of  each  specific  program  on  student  outcomes  and  on  student  and 
teacher  perceptions  of  school  climate. 

The  primary  research  questions  for  the  evaluation  were  as  follows: 

1 . What  is  the  average  effect  of  the  seven  universal,  school-based,  social  and  character  development 
programs  on  social  and  character  development  instruction  in  the  schools? 


3 The  What  Works  Clearinghouse  was  established  in  2002  by  the  U.S.  Department  of  Education’s  Institute  of  Education 
Sciences  to  assess  the  rigor  of  research  evidence  on  the  effectiveness  of  educational  interventions. 
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Figure  1.1.  Conceptual  model  for  social  and  character  development  interventions 


CS 


Intervention 

Program  structure 

Such  as 
Targeting  unit 
Classroom 
Entire  school 

Other  (after  school,  family) 

Curriculum  structure 

Distinct  activities 
Embedded  in  curriculum 

Activities  to  address  SACD  goals 

Such  as 

Violence  prevention/peace  promotion 
Social  and  emotional  development 
Character  education 
Tolerance  and  diversity 
Risk  prevention  and  health  promotion 
Civic  responsibility/community  service 
Behavior  management 


Social  Emotional  Competence 

Attitudes  about  aggression 

Self-efficacy 

Empathy 


► 


Behavior 


Positive  behavior 

Responsible  behavior 
Prosocial  behavior 
Self-regulation 
Cooperation 


Negative  behavior 

Aggression 
Minor  delinquency 
Disruptive  classroom  behavior 


► 


► 


Perceptions  of  School  Climate 

School  connectedness 
Victimization 

Feelings  of  safety  at  school 
Parent  involvement 


► 


► 


Academics 

Academic  competence 
School  engagement 
Grades 

Standardized  test  scores 


Moderating  Factors 

Child 

Family 

Community 

Gender 

Parenting  practices 

Community  risk  factors 

Socioeconomic  status 

Home  atmosphere 

Social  capital 

Race/ethnicity 

Risk  status 

Program 

School 

Prior  test  scores/grades 

Fidelity 

Activities  to  promote  social 

Intervention  dosage 

Intensity 

and  character  development 

Organizational  structure 

Chapter  1 . The  Social  and  Character  Development  Multiprogram  Evaluation 


Chapter  1 . The  Social  and  Character  Development  Multiprogram  Evaluation 


2.  What  is  the  average  effect  of  the  seven  universal,  school-based,  social  and  character  development 
programs  on  students’  social  and  emotional  competence,  behavior,  and  academics,  and  on 
perceptions  of  school  climate? 

3.  What  is  the  average  effect  of  each  specific  social  and  character  development  program  on 
students’  social  and  emotional  competence,  behavior,  and  academics,  and  on  perceptions  of 
school  climate? 

As  noted  in  figure  1 . 1 , a set  of  moderating  factors  was  expected  to  affect  the  success  of  any  SACD  program. 
The  potential  impact  of  these  factors  raised  secondary  research  questions  concerning  the  differential  impact 
of  a SACD  program  on  student  subgroups  and  by  the  quality  of  implementation  of  the  program. 

Subgroups  of  students  may  differ  enough  in  their  behaviors  that  the  impact  of  a SACD  program  may  have 
differential  impacts  on  those  subgroups.  Significant  gender  differences  exist  in  the  frequency  of  cooperative, 
prosocial  behaviors  and  problem  behaviors  among  young  children.  Aggression  and  hyperactive  behaviors 
tend  to  be  more  common  among  boys.  Among  violence  prevention  programs  that  have  examined  whether 
programs  are  differentially  effective  for  boys  and  girls,  four  have  indicated  significant  effects  for  boys  but  not 
for  girls  (Farrell  and  Meyer  1997;  Flay  et  al.  2004;  Kellam  et  al.  1998).  Given  these  observed  differences,  the 
average  effects  of  the  seven  programs  were  expected  to  differ  by  student  gender. 

Previous  research  has  also  documented  that  certain  family  and  community  characteristics  are  associated  with 
greater  prevalence  of  behavioral  problems  for  children.  Regarding  the  family,  children  from  poor  families  are 
at  greater  risk  for  behavioral  or  emotional  problems  (Brooks-Gunn  and  Duncan  1997)  and  show  difficulties 
with  aspects  of  social  competence  such  as  self-regulation  and  impulsivity  (Takeuchi,  Williams,  and  Adair 
1991).  Poor  parental  supervision  and  monitoring  are  associated  with  the  development  of  problem  behaviors 
in  children  (Gorman-Smith  et  al.  1996),  and  parental  involvement  has  been  linked  to  children’s  overall 
behavior  in  school,  motivation  to  learn,  grades  and  test  scores,  and  long-term  achievement  (Griffith  1996; 
Grolnick,  Ryan  and  Deci  1991;  Shaver  and  Walls  1998;  Zellman  and  Waterman  1998).  At  the  community 
level,  disorganization  (Sampson  2000)  and  exposure  to  community  violence  (Schwartz  and  Proctor  2000)  are 
associated  with  greater  risk  for  behavior  problems.  Neighborhood  disadvantage  has  also  been  consistently 
linked  with  increased  levels  of  child  aggression,  emotional  problems,  inattention  problems,  and  conduct 
problems  (Caspi  et  al.  2000;  Gershoff  and  Aber  2006).  As  a consequence,  the  average  effect  of  the  seven 
programs  was  expected  to  differ  for  children  experiencing  different  levels  of  family  and  community  risk. 

Early  childhood  conduct  problems  and  aggressive  behaviors  are  another  risk  factor  for  later  problem 
behaviors  such  as  fighting,  delinquency,  and  drug  involvement  (Dodge  and  Pettit  2003).  Meta-analytic  studies 
that  have  examined  the  effect  of  school-based  programs  on  problem  behaviors  indicate  larger  effect  sizes  for 
programs  targeting  “high-risk”  populations  than  general  school  populations  (Wilson,  Gottfredson,  and 
Najaka  2001;  Wilson,  Lipsey,  and  Derzon  2003). 

The  differential  behavior  found  among  certain  student  subgroups  and  some  evidence  of  differential  responses 
by  these  subgroups  to  programs  led  to  the  fourth  research  question: 

4.  Do  the  average  effects  of  the  seven  universal,  school-based  social  and  character  development 
programs  differ  by  (a)  students’  gender  and  (b)  students’  initial  risk  factors  (socioeconomic,  family, 
community,  and  earlier  child  behavior)? 

Fidelity  of  program  implementation  has  been  found  to  strongly  affect  the  impacts  of  interventions  of  all 
types.  A number  of  studies  show  that  programs  delivered  with  high  fidelity  lead  to  better  outcomes  than 
programs  implemented  with  lower  fidelity  (e.g.,  Crandall  et  al.  1982;  Stringfield  et  al.  1997).  This  raised  a fifth 
research  question: 

5.  In  the  treatment  schools,  is  there  an  association  between  the  level  of  implementation  of  the  social 
and  character  development  programs  and  impacts  on  student  outcomes? 
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The  Social  and  Character  Development  Research  Program 

In  2003,  IES,  in  collaboration  with  DVP,  conducted  a peer-reviewed  research  grant  competition  in  which 
applicants  proposed  to  (1)  implement  a universal,  school-based  intervention  for  elementary  school  children 
intended  to  promote  positive  behaviors  and  attitudes  and/or  reduce  negative  or  antisocial  behaviors  and 
attitudes  and  (2)  evaluate  the  intervention  using  a cluster  randomized  design  in  which  schools  would  be  the 
unit  of  assignment.  Applicants  were  encouraged  to  both  identify  measures  that  assessed  the  specific  attitudes 
and  behaviors  that  their  proposed  intervention  was  designed  to  improve  and  to  take  into  account  the  fact  that 
the  implementation  and  evaluation  of  the  intervention  would  take  place  within  the  context  of  a larger 
multiprogram  evaluation  in  which  a third  party  would  collect  comparable  outcomes  from  all  schools 
participating  in  the  evaluation.  That  is,  in  addition  to  whatever  measures  the  applicants  proposed  to  use,  all  of 
the  interventions  in  the  SACD  Research  Program  would  be  evaluated  on  a set  of  measures  that  would  be 
collected  by  an  evaluation  team  that  would  be  separate  from  the  research  teams  involved  in  implementing  the 
interventions. 

Seven  research  teams  were  selected  through  a peer-review  process,  and  they  entered  into  cooperative 
agreements  with  IES  to  randomize  schools  to  either  implement  an  elementary  school  SACD  program  or 
continue  with  their  existing  educational  practice.  Each  research  team  evaluated  one  program  in  its  chosen  site 
by  following  one  cohort  of  students  from  third  through  fifth  grades,  with  the  treatment  students  receiving  the 
program  in  all  3 years  (students  in  the  other  elementary  grades  in  the  treatment  schools  also  received  the 
program  but  were  not  part  of  the  evaluation).  Six  of  the  programs  had  separate  curriculum  units  for  each 
grade,  and  one  program  (ABC)  had  a single  standard  unit  with  modifications  for  each  grade.  Programs  were 
not  randomized  across  teams  or  their  sites;  schools  were  randomized  to  treatment  or  control  conditions 
within  a team  and  its  site.  The  six  panels  of  table  1 . 1 identify  each  program  and  its  research  team4  and  sites; 
describe  general  program  characteristics,  strategies,  and  type  and  amount  of  professional  development 
provided;  outline  the  SACD  goals  addressed  by  program  activities;  and  note  the  outcomes  addressed  by  the 
evaluation.  The  key  point  here  is  that  the  panels  demonstrate  how  the  seven  programs  drew  from  a common 
set  of  characteristics,  techniques,  goals,  and  outcomes,  while  differing  in  which  of  these  each  program  used  or 
addressed.  For  example,  Panel  2 shows  that  four  programs  included  behavior  management  while  three  did 
not,  and  that  all  provided  either  a manual  or  scripted  lessons  but  that  programs  differed  in  the  degree  of 
teacher  adaptation.  Panel  3 shows  that  programs  used  many  of  the  same  activities  and  often  addressed  similar 
content  (e.g.,  problem  solving),  but  each  program  had  specific  content  that  it  alone  covered.  Panel  4 shows 
that  all  programs  used  the  combination  of  pre-implementation  training  plus  ongoing  consultation  but  that 
training  time,  nonteacher  personnel  included,  and  frequency  and  method  of  consultation  varied  among  them. 
Panels  5 and  6 note  that  some  programs  addressed  all  the  SACD  goals  and  outcomes  (described  later  in  this 
chapter)  while  others  focused  on  a selection  of  these.  More  detailed  descriptions  of  the  programs  are  included 
in  the  program-specific  chapters  of  this  report. 


4 Members  of  two  of  the  seven  research  teams  had  developed  the  interventions  they  were  evaluating  (University  at 
Buffalo,  The  State  University  of  New  York,  and  the  University  of  North  Carolina  at  Chapel  Hill),  and  one  team  included 
a developer  of  its  intervention  (Oregon  State  University). 
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Panel  1 : The  seven  programs 


Program/Source 

Features 

Curriculum 

structure 

Research  team 

Site 

Academic  and  Behavioral  Competencies  Program 
Center  for  Children  and  Families 
University  at  Buffalo,  State  University  of  New  York 

Social  skills  training  and  behavior 
management 

Classroom 
curriculum  and 
targeted 
component 

University  at 
Buffalo,  State 
University  of 
New  York 

Buffalo,  New  York, 
and  two  charter 
schools 

Competence  Support  Program 
School  of  Social  Work 
University  of  North  Carolina-Chapel  Hill 

Social  and  emotional  learning,  social 
dynamics  training,  and  behavior 
management:  social  information 
processing,  social  problem  solving, 
peer  networks 

Classroom 
curriculum  and 
intensive  teacher 
training 

University  of 
North  Carolina 
at  Chapel  Hill 

Hoke  and  Wayne 
Counties,  North 
Carolina 

Love  In  a Big  World 
Love  In  a Big  World 
Nashville,  Tennessee 

Character  education:  courage,  honesty, 
kindness,  caring 

Classroom 
curriculum  and 
whole-school 
approach 

Vanderbilt 

University 

Maury  and 
Murfreesboro 
Counties, 
Tennessee 

Positive  Action 

Positive  Action,  Inc. 
Twin  Falls,  Idaho 

Social  and  emotional  learning:  values, 
empathy,  self-control,  social  skills, 
social  bonding,  self-efficacy,  honesty, 
goal  setting 

Classroom 
curriculum  and 
whole-school 
approach 

Oregon  State 
University 

Chicago,  Illinois 

Promoting  Alternative  Thinking  Strategies 
Channing  Bete  Company 
South  Deerfield,  Massachusetts 

Social  and  emotional  learning: 
emotional  literacy,  self-control,  social 
competence,  peer  relations, 
interpersonal  problem  solving 

Classroom 

curriculum 

The  Children’s 
Institute 

Robbinsdale, 
Minnesota,  and 
Rochester  and 
Rush-Henrietta, 
New  York 

The  4Rs  Program  (Reading,  Writing,  Respect,  and 
Resolution) 

Morningside  Center  for  Teaching  Social  Responsibility 
New  York,  New  York 

Conflict  resolution  and  literacy:  social 
problem  solving,  anger  management, 
mediation 

Classroom 

curriculum 

New  York 
University 

New  York  City, 
New  York 

Second  Step 

Committee  for  Children 
Seattle,  Washington 

Violence  prevention  and  social  and 
emotional  learning:  empathy,  anger 
management,  impulse  control,  and 
problem  solving 

Classroom 

curriculum 

University  of 
Maryland, 
College  Park 

Anne  Arundel 
County,  Maryland 

See  notes  at  end  of  table. 
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Table  1.1.  Social  and  Character  Development  programs — Continued 

Panel  2:  General  characteristics 


Target 

population 

Program  components 

Level  of 

Flexibility 

Program 

Peer 

Parent 

Classroom 

Schoolwide 

Community 

Training 

integration 

Manualized 

Adaptability 

ABC 

Universal 

and 

targeted 

In  class 
and  out 
of  class 

Contact 

Lessons  and 

behavior 

management 

Planned 

events,  reward 

programs, 

schoolwide 

rules, 

discipline 

policies 

None  or 
not  major 
focus 

Pretraining 

and 

ongoing 

Add-on 

curriculum 

and 

schoolwide 

activities 

Manual 
includes 
modules  for  all 
program 
components 

Program  may 
be 

individualized 
by  school, 
classroom,  and 
student 

CSP 

Universal 

In  class 

No  contact 

Lessons, 
behavior 
management, 
and  social 
dynamics 

None  or  not 
major  focus 

None  or 
not  major 
focus 

Pretraining 

and 

ongoing  for 

teachers 

and 

counselors 

Add-on 

curriculum 

Scripted 
lessons  for 
teachers 

Teachers  may 
adapt  program 
to  setting 

LBW 

Universal 

In  class 
and  out 
of  class 

Contact  and 
involvement 

Lessons  and 

behavior 

management 

Planned 

events, 

modeling, 

program 

artifacts 

None  or 
not  major 
focus 

Pretraining 

and 

ongoing 

Add-on 

curriculum 

and 

schoolwide 

activities 

Curriculum 

guidebook 

Less  adaptable 

PATHS 

Universal 

In  class 

Contact  and 
involvement 

Lessons 

Planned 

events, 

program 

artifacts 

None  or 
not  major 
focus 

Pretraining 

and 

ongoing 

Add-on 

curriculum 

and 

schoolwide 

activities 

Scripted 

curriculum 

guidebook 

Less  adaptable 

See  notes  at  end  of  table. 
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Table  1.1.  Social  and  Character  Development  programs — Continued 

Panel  2:  General  characteristics — Continued 


Target 

population 

Program  components 

Level  of 

Flexibility 

Program 

Peer 

Parent 

Classroom 

Schoolwide 

Community 

Training 

integration 

Manualized 

Adaptability 

PA 

Universal 

In  class 
and  out 
of  class 

Training, 

contact, 

and 

involvement 

Lessons  and 

behavior 

management 

Planned  events, 
includes 
program 
artifacts  PA 
coordinator  and 
PA  committee 

None  or  not 
major  focus 

Pretraining 

and 

ongoing 

Add-on 

curriculum 

and 

schoolwide 

activities 

Curriculum 
guidebook  with 
scripted  lessons 
and  materials, 
schoolwide 
guidebooks, 
parent  manual, 
and  support 
staff  manual 

Program  staff 
and  principal 
adapt  activities 
to  needs  and 
setting 

4Rs 

Universal 

In  class 

Contact 

and 

involvement 

Lessons 

None  or  not 
major  focus 

None  or  not 
major  focus 

Pretraining 

and 

ongoing 

Core 

curriculum 

(language 

arts) 

Curriculum 

guidebook 

Less  adaptable 

SS 

Universal 

In  class 

Contact 

Lessons 

Program 
artifacts  and 
modeling 

None  or  not 
major  focus 

Pretraining 

and 

ongoing 

Add-on 

curriculum 

Curriculum 

guidebook 

Character 
development 
planning  teams 

See  notes  at  end  of  table. 
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Table  1 .1 . Social  and  Character  Development  programs — Conti 

Panel  3:  Description  of  strategies 


Who 

Program  delivers 
ABC  Teacher 


Who  Activities 
delivers  and  tools 
Teacher  Social  skill  of 
the  day 

description  and 
classroom  or 
schoolwide 
role-play 


Lessons 

Content 
Social  skills, 
problem  solving, 
classroom 
management, 
peer 

relationships 


Classroom 


Frequency 
Daily  5-10 
minutes  plus 
reinforcement; 
twice  a week 
45-minute  peer 
program 


CSP 


LBW 


Teacher  Story  reading,  Problem  45  minutes, 

discussion,  solving,  emotion  once  per 

role-playing,  identification  week;  28 

worksheets,  and  regulation,  lessons 

games,  artifacts  identifying 

(e.g.,  turtle  social  cues, 

puppet),  art  goal  formation, 

projects  choosing 

options,  and 

behavior 

regulation 


Teacher  Story  reading,  Character  traits  Daily  10- to 

writing,  and  moral  15-minute 

interdisciplinary  virtues  lessons,  30 

activities,  weeks 

rewards,  singing 


See  notes  at  end  of  table. 


Strategies 


Who 

Activities 

Supplement  to 

Schoolwide 

delivers 

and  tools 

Frequency 

classroom 

activities 

Teacher 

Recognition 

Daily 

Homework, 

Schoolwide 

and 

peer 

rules,  recording 

reinforcement 

mediation,  and 

of  rule  violations, 

of  good 

parenting 

positive 

behavior 

program 

affirmation, 

(e.g.,  positive 

afterschool 

notes, 
behavior 
reports,  time 
out,  fun 
activities) 

program 

Teacher 

Prevention  of 
social 
hierarchies 
that  promote 
conflict;  social 
reinforcement 
for  positive 
behavior 

Daily 

None 

None 

Teacher  Recognition  Daily 

Parent 

Weekly 

and 

newsletters 

announcements; 

reinforcement 

occasional 

of  good 

assemblies;  two 

behavior; 

service  projects; 

Modeling  of 

program  artifacts 

character 

traits 

Chapter  1.  The  Social  and  Character  Development  Multiprogram  Evaluation 


Table  1.1.  Social  and  Character  Development  programs — Continued 

Panel  3:  Description  of  strategies — Continued 


Classroom 

Lessons 

Strategies 

Who 

Activities 

Who 

Activities 

Supplement  to 

Schoolwide 

Program 

delivers 

and  tools 

Content 

Frequency 

delivers 

and  tools 

Frequency 

classroom 

activities 

PATHS 

Teacher 

Direct 

Emotion 

20-30 

Teacher 

Modeling  of 

Daily 

Parent 

End-of-year 

instruction, 

understanding 

minutes  per 

skills 

newsletter 

PATHS  party 

storytelling, 

and  control, 

day,  3-5  days 

and 

and  program 

discussion, 

behavior 

per  week 

engagement 

artifacts 

role-playing, 

regulation, 

in  homework 

utilization  of 

problem  solving, 

artifacts  (e.g., 
posters,  turtle 
puppet), 
worksheets 

making  friends 

PA 

Teacher 

Direct 

Self-concept, 

15-  to  20- 

Teacher 

Teaches, 

Daily 

Parent 

Occasional 

instruction, 

physical  health, 

minute 

practices, 

newsletters; 

assemblies; 

story  reading, 

intellectual 

lessons,  4 

recognizes, 

parent  manual 

service  projects; 

writing,  role- 

growth, 

days  per 

and  models 

with  PA 

PA  days  and 

playing, 

emotional  and 

week 

positive 

activities, 

year-end  event; 

discussion, 

behavioral  self- 

actions with 

letters,  and 

Principal  climate 

singing,  games, 

regulation, 

curriculum 

strategies; 

program  with 

worksheets, 

getting  along  with 

and  climate 

and  parent 

reinforcement 

puppets,  plays 

others  (social 

activities  and 

night 

recognition 

and  poems 

skills),  self- 
honesty, self- 
improvement, 
goal  setting, 
character  traits 

materials 

activities 

See  notes  at  end  of  table. 
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Table  1.1.  Social  and  Character  Development  programs — Continued 

Panel  3:  Description  of  strategies — Continued 


Classroom 

Lessons 

Strategies 

Program 

Who 

delivers 

Activities 
and  tools 

Content 

Frequency 

Who 

delivers 

Activities 
and  tools 

Frequency 

Supplement  to 
classroom 

Schoolwide 

activities 

4Rs 

Teacher 

Story  reading, 
role-playing, 
discussions, 
reflections, 
sharing 
exercises, 
brain- 
storming, 
songs, 
worksheets 

Literacy  focus; 
building 
community, 
feelings, 
listening, 
assertiveness, 
problem  solving, 
diversity,  making 
a difference 

1-hour 
lessons  at 
least  once 
per  week; 
31  lessons 

Teacher 

Modeling  of 
skills 

Daily 

Parent-child 
connections 
homework  and 
parent 
workshops 

None 

SS 

Teacher 

Storytelling, 

discussion, 

videos, 

role-playing, 

anticipation, 

recall, 

learning 

points 

Empathy, 
impulse  control, 
problem  solving, 
anger 

management 

30-minute 

lessons, 

1 -2  days 
per  week 

Teacher 

Modeling  of 
and 

reinforcement 
of  skills 

Weekly 

lessons: 

daily 

generalization 

Take-home 
letters, 
homework, 
family  guide 
and  video 

Program  artifacts, 
training  of  school 
staff 

See  notes  at  end  of  table. 
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Table  1.1.  Social  and  Character  Development  programs — Continued 

Panel  4:  Professional  development 


Pre-implementation 

Ongoing  consultation 

Teachers 

Other 

Teachers 

Other 

Program 

Content 

Duration 

Content 

Duration 

Content 

Duration 

Content 

Duration 

ABC 

Behavioral 

management 

techniques, 

schoolwide  rules, 

discipline 

policies, 

classroom 

management 

policies 

9 hours 

Staff  receive 
teacher  training 

9 hours 

Teacher 
consultation, 
and  coaching 
on  use  of 
classroom 
management 

Monthly  or 
more 

frequently  per 
teacher  or 
school 
request 

School 

consultation  on 
policies  and 
procedures 

As  needed,  with 
regular  follow- 
up 

CSP 

Training  on 

curriculum 

implementation, 

behavior 

management, 

and  social 

dynamics 

4 hours 

Principal 
consultation  on 
development  of 
schoolwide 
management 
policy 

As  requested 

Consultations 
on  use  of 
program 
materials 

2 times 
monthly 

None 

None 

LBW 

Workshop  on 
lesson  plans  and 
logistics  of 
program 
implementation 

1 day;  3 hours 

Principal  and 
staff;  Same  as 
teacher  training 

1 day;  3 hours 

Faculty 
boosters  with 
reminders 
about  program 
implementation 
and  discussion 
of  challenges 

24  weekly 
sessions; 
biannual 
meeting 

Program 
coordinator; 
calls  with 
program 
developer  to 
improve 
schoolwide 
implementation 

Once  every  6 
months 

See  notes  at  end  of  table. 
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Table  1.1.  Social  and  Character  Development  programs — Continued 

Panel  4:  Professional  development — Continued 


Pre-implementation 

Ongoing  consultation 

Teachers 

Other 

Teachers 

Other 

Program 

Content 

Duration 

Content 

Duration 

Content 

Duration 

Content 

Duration 

PATHS 

Training  on 

2 days 

Principal  and 

2 days 

Technical 

Weekly 

Principal  and 

Biweekly  calls; 

concepts, 

school  mental 

assistance  and 

consultations; 

school  staff; 

2-day  summer 

curriculum 

health  staff; 

implementation 

2-day  summer 

model  lessons, 

meeting 

implementation, 

same  as 

consultation, 

meeting 

team  teaching, 

how  to  integrate 

teacher  training 

individual 

general 

activities  with 

meetings  with 

feedback; 

traditional 

each  teacher, 

planning  for 

instruction 

attendance  at 
grade-level 
meetings, 
planning  for 
following  year 

following  year 

PA 

Training  on 

1/2  day 

Principal 

1/2  day 

Group  session 

Up  to  1/2  day 

Meeting  of 

One  session  of 

concepts  and 

training  on 

to  provide 

once  per  year, 

principals  and 

3 hours  per 

delivery  of 

appointing 

technical 

and  up  to  1/2 

coordinators  to 

year 

curriculum, 

committees 

assistance  and 

day  visits  to 

provide 

school  climate 

and 

share 

schools 

technical 

activities  to 

coordinating 

experiences 

monthly 

assistance  and 

reinforce  positive 

school  climate 

and  challenges; 

share 

behaviors  and 

activities  with 

visits  for 

successes  and 

parent 

family  groups 

technical 

challenges 

involvement 

support 

See  notes  at  end  of  table. 
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Table  1.1.  Social  and  Character  Development  programs — Continued 

Panel  4:  Professional  development — Continued 


Pre-implementation 

Ongoing  consultation 

Teachers 

Other 

Teachers 

Other 

Program 

Content 

Duration 

Content 

Duration 

Content 

Duration 

Content 

Duration 

4Rs 

Course  to  learn 
curriculum, 
improve  on  own 
skills,  create  a 
vision  of 
community,  set 
consistent  rules, 
behavior 
management 

25  hours 

None 

None 

Training  and 
support  in 
mediation, 
negotiation, 
role-playing, 
class  meetings, 
behavior 
management 

Individualized 
coaching  12 
times  per  year; 
3-day  summer 
institute 

None 

None 

SS 

Training  on  child 
development, 
social  skills,  and 
curriculum 
delivery 

2 days 

Character 
development 
planning  team, 
school 

counselor,  and 
other  staff; 
training  on  child 
development, 
social  skills,  and 
curriculum 
content 

2.5  hours 

Year-end 
workshop  to 
review  student 
outcome  data 
and  plan  for 
following  year 

1 day 

School 
counselor 
guidance  on 
strategies  and 
challenges; 
principal  and 
staff  review  of 
student 
outcome  data 
and  plan  for 
following  year 

1/2  day 
counselor 
meetings  about 
monthly;  1-day 
year-end 
workshop 

See  notes  at  end  of  table. 
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Table  1.1.  Social  and  Character  Development  programs — Continued 

Panel  5:  Activities  to  promote  the  six  SACD  goals  and  behavior  management 


Program 

Violence  prevention 
and  peace 
promotion 

Social  and  emotional 
development 

Character  education 

Tolerance  and 
diversity 

Risk  prevention  and 
health  promotion 

Civic  responsibility 
and  community 
service 

Behavior 

management 

ABC 

s 

✓ 

✓ 

CSP 

✓ 

✓ 

s 

LBW 

✓ 

✓ 

✓ 

✓ 

PATHS 

s 

✓ 

s 

✓ 

s 

✓ 

V 

PA 

V 

✓ 

s 

✓ 

V 

✓ 

V 

4Rs 

V 

✓ 

✓ 

✓ 

s 

✓ 

s 

SS 

✓ 

✓ 

✓ 

See  notes  at  end  of  table. 
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Table  1.1.  Social  and  Character  Development  programs — Continued 

Panel  6:  SACD  outcomes  addressed 


Program 

Self-Efficacy 
for  Peer 
Interactions 

Normative 
Beliefs  About 
Aaqression 

Empathy 

Altruistic 

Behavior 

Positive 

Social 

Behavior 

Problem 

Behavior 

Engage-  Academic 
ment  with  Competence 
Learninq  and  Motivation 

Positive 

School 

Orientation 

Negative 

School 

Orientation 

Student 
Afraid  at 
School 

Victimization 
at  School 

Student 

Feelings  Support  for 
of  Safety  Teachers 

ABC 

V 

V 

s 

CSP 

V 

V 

V 

s 

V 

V 

LBW 

V 

V 

V 

✓ 

PA 

V 

V 

V 

V 

V 

V 

V 

V 

PATHS 

V 

V 

V 

V 

V 

V 

✓ 

V 

4Rs 

V 

V 

V 

V 

V 

V 

V 

SS 

V 

V 

V 

V 

NOTE:  Abbreviations  are 

ABC:  Academic  and  Behavioral  Competencies  Program 
CSP:  Competence  Support  Program 
LBW:  Love  In  a Big  World 
PA:  Positive  Action 

PATHS:  Promoting  Alternative  Thinking  Strategies 

4Rs:  The  4Rs  Program  (Reading,  Writing,  Respect,  and  Resolution) 

SS:  Second  Step 

v : Outcome  addressed 

Blank  cell:  Outcome  not  addressed 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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The  commonalities  and  differences  among  the  seven  SACD  programs  are  important  for  several  reasons.  They 
make  clear  that  there  is  no  single  SACD  universal  schoolwide  program,  but  a set  of  them  that  are  in  many 
ways  similar  but  also  different  enough  to  distinguish  them.  The  commonalities  justify  this  study’s  evaluation 
approach  of  both  combining  results  from  all  seven  programs  and  examining  the  results  from  each  program 
separately  for  a common  set  of  outcome  measures  for  measuring  students’  social  and  character  development. 
At  the  same  time,  it  is  important  to  keep  in  mind  the  differences  when  considering  the  results  for  each  of  the 
seven  programs. 

To  standardize  the  evaluation,  a competitive  peer-review  process  was  used  to  select  a third  party  to  collect 
and  analyze  the  common  set  of  data  from  all  the  schools.  MPR  was  selected  for  this  role. 

As  in  most  evaluations  of  education  interventions,  the  SACD  programs  were  not  compared  to  a no-treatment 
control.  The  control  schools  continued  to  implement  the  social  and  character  development  activities  that 
constituted  their  standard  practice.  A number  of  researchers  have  observed  that  schools  commonly  use 
support  services,  intervention  curricula,  and  discipline  management  strategies  to  promote  social  and  character 
development  and  prevent  problem  behavior  (e.g.,  Crosse  et  al.  2001;  Gottfredson  and  Gottfredson  2001),  and 
in  many  states,  schools  are  required  by  law  to  implement  such  programs  and  practices.  For  example,  under 
the  Student  Citizen  Act  of  2001,  character  education  was  made  part  of  the  standard  curriculum  for  North 
Carolina  schools  starting  in  the  2002-03  school  year;  in  New  York,  the  Safe  Schools  Against  Violence  in 
Education  Act  of  2001  encouraged  districts  to  promote  instruction  in  civility,  citizenship,  and  character 
education;  and  Illinois’  Children’s  Mental  Health  Act  required  all  school  districts  to  develop  a policy  for 
incorporating  social  and  emotional  competence  into  the  district’s  educational  program  by  August  31,  2004. 
For  this  reason,  data  were  collected  on  the  implementation  of  social  and  character  development  activities 
from  all  schools.  These  data  provide  contextual  information  on  the  prevalence  of  social  and  character 
development  activities  occurring  in  both  the  treatment  and  control  schools. 

Study  Design  and  Methodology 

The  SACD  multiprogram  evaluation  was  a 3-year  longitudinal  study  following  one  cohort  of  students  from 
third  through  fifth  grades.5  It  was  designed  to  determine  the  effects  of  seven  SACD  programs  on  the  social 
and  character  development  of  students  attending  the  schools  in  which  the  interventions  were  implemented. 
This  design  stems  in  part  from  the  programs  being  chosen  by  the  quality  of  the  grant  applications,  so  they  are 
not  necessarily  representative  of  the  full  set  of  universal  school-based  SACD  interventions.  The  results,  then, 
are  not  generalizable  beyond  the  seven  programs. 

The  evaluation  examined  the  average  effect  of  the  programs  on  a common  set  of  student  outcomes  including 
social  and  emotional  competence,  behavior,  academics,  and  perceptions  of  school  climate.  Initial  data  were 
collected  on  consenting  students  in  the  third  grade  during  fall  2004.  Follow-up  data  were  collected  at 
subsequent  time  points:  (1)  in  spring  2005  from  third-grade  students,  (2)  in  fall  2005  when  the  original  third- 
grade  cohort  began  fourth  grade,  (3)  in  spring  2006  when  the  original  third-grade  cohort  completed  fourth 
grade,  and  (4)  in  spring  2007  when  the  original  third-grade  cohort  completed  fifth  grade.  Data  for  the  study 
included  survey  data  from  students,  students’  primary  caregivers,  teachers,  and  interview  responses  from 
principals. 


5 The  multiprogram  evaluation  also  included  12  additional  schools  recruited  in  the  second  year  of  the  study.  This  second 
cohort  of  third-grade  students  was  followed  for  2 years  through  the  fourth  grade.  Initial  data  were  collected  in  fall  2005, 
and  follow-up  data  in  spring  2006  and  spring  2007.  These  schools  were  recruited  by  the  Children’s  Institute  (»  = 4), 
University  at  Buffalo,  The  State  University  of  New  York  (»  = 2),  University  of  North  Carolina  at  Chapel  Hill  (»  = 4), 
and  Vanderbilt  University  (»  = 2).  Appendix  A describes  the  combined  analysis  of  Cohorts  1 and  2.  The  Executive 
Summary  and  chapters  1-8  describe  the  analysis  of  Cohort  1 only. 
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Sample  and  Random  Assignment 

Schools  were  the  unit  of  assignment,  in  part  because  the  programs  were  applied  at  the  school  level  (or  at  least 
in  multiple  grades  in  a school).  As  a result,  contamination  from  the  treatment  group  to  the  control  group  was 
unlikely  because  of  the  physical  separation  of  schools.  From  each  school,  data  were  collected  from  students  in 
grades  3,  4,  and  then  5 in  consecutive  years.  Data  were  also  collected  from  students’  primary  caregivers,  from 
the  principals  of  the  schools,  and  from  the  teachers  in  all  three  grades.  Parents  or  primary  caregivers, 
principals,  and  teachers  all  had  to  give  their  consent  before  taking  part  in  the  study.  Students  gave  assent  after 
their  primary  caregivers  had  provided  consent.  Beginning  in  the  spring  and  early  fall  of  2004,  consent  forms 
were  sent  home  with  students  for  primary  caregivers  to  review  and  sign,  if  desired,  indicating  their  consent 
for  their  own  participation  in  the  study  and  for  the  student’s  participation  in  the  study.  The  consent  process 
continued  at  each  follow-up  for  students  who  were  new  entrants  to  the  schools,  or  who  did  not  return  a 
consent  form  in  fall  2004.  During  each  data  collection,  students  themselves  were  asked  whether  they  would 
assent  to  participate  in  the  study.  The  research  teams  carried  out  the  consent  process  and  other  procedures 
concerning  human  subjects  after  their  review  and  approval  by  institutional  review  boards  at  each  participating 
institution  and  by  the  Public/Private  Ventures  Institutional  Review  Board. 

Each  research  team  set  the  eligibility  criteria  for  schools  within  its  site  and  recruited  10  to  14  schools  for  the 
study.6  In  general,  schools  were  purposively  selected  based  on  (1)  their  lack  of  schoolwide  institutionalized 
programs  addressing  social  and  character  development,  (2)  their  willingness  to  participate  in  random 
assignment  and  engage  in  data  collection  activities,  and  (3)  their  ability  to  implement  the  program  schoolwide 
(all  elementary  grades)  if  assigned  to  the  treatment  condition.  Additional  eligibility  criteria  employed  by 
individual  teams  included  relatively  low  performance  on  standardized  tests,  a relatively  high  percentage  of 
students  eligible  for  free  and  reduced-price  lunch,  low  student  mobility  rates,  and  a sufficient  number  of 
students  per  school  to  meet  sample  size  requirements. 

The  research  teams  recruited  a total  of  84  schools  into  the  study  beginning  with  the  initial  data  collection  in 
fall  2004  (table  1.2).  Eighty- three  schools  completed  the  evaluation,  with  one  (a  control  school)  dropping  out 
before  the  start  of  the  second  year  after  converting  to  a magnet  school.  Some  schools  changed  their  grade 
structures  over  time  but  in  only  one  school  did  this  lead  to  the  students  in  the  study  not  receiving  the 
intervention  (the  loss  of  students  in  fifth  grade  at  one  CSP  school).  In  Year  1,  the  highest  grade  was  fourth  or 
fifth  grade  at  46  schools,  sixth  or  seventh  grade  at  18  schools,  and  eighth  grade  at  20  schools.  In  Years  2 and 
3,  these  figures  were  47  and  49  schools  having  fourth  or  fifth  grade  as  the  highest  grade;  15  and  12  schools 
with  sixth  or  seventh  grade;  and  21  and  22  schools  with  eighth  grade,  respectively. 

After  recruiting  the  schools,  each  research  team  created  five  to  seven  matched  pairs  of  schools.  Within  each  pair, 
one  school  was  randomly  assigned  to  the  treatment  group  and  the  other  to  the  control  group.  Stratified  sampling 
was  used  to  guard  against  the  possibility  that  random  assignment  of  a small  number  of  schools  could  produce  sets 
of  treatment  and  control  schools  that  had  markedly  different  characteristics.  An  algorithm  was  developed  to  select 
the  best  pairs  by  minimizing  tire  distance  between  several  measurable  characteristics  for  schools  within  each  pair.  A 
set  of  candidate  pairs  was  selected  such  that  the  overall  quality  of  matches  across  all  the  schools  remained  as  high 
as  possible,  without  creating  any  serious  mismatches  for  any  subset  of  tire  individual  pairs.  Each  team  executed  tire 
matching  algorithm  with  somewhat  different  school  characteristic  variables,  depending  on  what  data  were 
available,  and  exercised  its  best  judgment,  based  on  knowledge  of  the  schools  involved,  as  to  which  candidate 
pairing  was  the  best  one.  After  the  best  pairing  was  established,  one  member  of  each  pair  was  randomly  selected  to 
be  in  the  treatment  group  and  tire  other  was  assigned  to  the  control  group. 


6 The  New  York  University  research  team  recruited  18  schools  but  only  14  were  included  in  the  multiprogram 
evaluation. 
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Table  1.2.  Multiprogram  evaluation  school  sample 


Year  1 

(Spring/Fall  3rd  grade) 

Year  2 

(Spring/Fall  4th  grade) 

Year  3 

(Spring/Fall  5th  grade) 

Program 

Total 

Treatment  Control 

Total 

Treatment  Control 

Total 

Treatment  Control 

Total  number  of  schools 

84 

42 

42 

83 

42 

41 

83 

42 

41 

ABC 

12 

6 

6 

12 

6 

6 

12 

6 

6 

CSP 

10 

5 

5 

10 

5 

5 

10 

5 

5 

LBW1 

12 

6 

6 

11 

6 

5 

11 

6 

5 

PA 

14 

7 

7 

14 

7 

7 

14 

7 

7 

PATHS 

10 

5 

5 

10 

5 

5 

10 

5 

5 

4Rs 

14 

7 

7 

14 

7 

7 

14 

7 

7 

SS 

12 

6 

6 

12 

6 

6 

12 

6 

6 

1 One  control  school  became  a magnet  school  and  dropped  out  of  the  evaluation. 


NOTE:  Abbreviations  are 

ABC:  Academic  and  Behavioral  Competencies  Program 
CSP:  Competence  Support  Program 
LBW:  Love  In  a Big  World 
i\o  PA:  Positive  Action 

^ PATHS:  Promoting  Alternative  Thinking  Strategies 

4Rs:  The  4Rs  Program  (Reading,  Writing,  Respect,  and  Resolution) 

SS:  Second  Step 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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The  teacher  sample  for  the  evaluation  included  the  teachers  in  the  grade  that  die  student  cohort  attended  in  order 
to  obtain  teacher  reports  on  students  and  all  the  third-,  fourth-,  and  fifth-grade  teachers  in  the  participating  schools 
during  all  3 school  years  in  order  to  measure  the  level  of  SACD  activities  in  the  schools.  The  principal  sample 
included  the  principals  at  each  of  the  84  schools. 

The  student  sample  for  the  multiprogram  evaluation  included  students  enrolled  in  the  third  grade  during  the  2004- 
OS  school  year,  students  enrolled  in  the  fourth  grade  during  die  2005-06  school  year,  and  students  enrolled  in  the 
fifth  grade  during  the  2006-07  school  year,  with  “enrollment”  defined  as  being  present  in  the  study  schools  at  the 
time  of  student  data  collection.  Thus,  the  sample  included  a cohort  of  third-grade  students  as  they  progressed 
through  their  fifth-grade  year.  In  addition,  it  included  students  who  entered  the  study  schools  over  the  course  of 
the  study  (i.e.,  students  who  entered  study  schools  as  third  graders  during  the  2004-05  academic  year,  or  as  fourth 
graders  in  the  2005-06  academic  year,  or  as  fifth  graders  in  the  2006-07  academic  year).  Thus,  the  specific  students 
in  the  sample  changed  between  the  beginning  of  data  collection  during  fall  2004  and  subsequent  rounds  of  data 
collection.  Up  to  the  time  of  a specific  data  collection  follow-up,  students  exposed  to  an  intervention  were  from 
one  of  three  groups: 

1 . Children  enrolled  in  third  grade  at  a study  school  in  fall  2004  and  during  tire  subsequent  rounds  of  data 
collection — original  sample  who  remained  at  the  schools  (stayers); 

2.  Children  enrolled  in  third  grade  at  a study  school  in  fall  2004  but  not  enrolled  during  a subsequent  data 
collection  period — original  sample  who  left  between  fall  2004  data  collection  and  a subsequent  data 
collection  (leavers);  and 

3.  Children  who  enrolled  in  the  grade  of  the  student  cohort  at  a study  school  after  the  fall  2004  data 
collection  but  before  the  subsequent  data  collection  initiative  (new  entrants). 

A total  of  6,567  third-graders  were  initially  enrolled  in  the  study  schools  in  fall  2004  (table  1.3).  Due  to  the 
combination  of  leavers  and  new  entrants,  the  total  was  6,597  in  spring  2005.  It  fell  to  6,415  in  spring  2006  and  to 
6,249  in  spring  2007.  Over  this  period  the  treatment  group  made  up  51  percent  to  52  percent  of  die  sample.  The 
average  number  of  students  per  school  ranged  from  75  to  81;  however,  the  actual  number  per  school  varied  across 
the  programs  from  23  to  1 64.  Over  time,  new  entrants  grew  as  a percentage  of  the  sample  from  6 percent  to  20 
percent  to  28  percent,  and  there  were  no  statistically  significant  differences  each  year  between  the  percentages  of 
new  entrants  in  the  treatment  and  control  groups.  The  percentage  of  the  original  sample  that  left  also  grew  from 
follow-up  to  follow-up,  going  from  6 percent  to  22  percent  to  32  percent,  again  with  no  statistically  significant 
differences  each  year  between  the  treatment  and  control  groups.  Leavers  were  not  tracked  once  they  left  the  study 
schools.7  That  is,  if  a third-grade  student  was  enrolled  in  a study  school  during  fall  2004  but  then  left  before  spring 
2005,  that  student  was  not  tracked  into  the  new  school  and  new  data  from  that  student  were  not  collected  in  spring 
2005.  As  a result  of  this  turnover,  69  percent  of  the  original  students  (stayers)  remained  in  the  study  and  they  made 
up  72  percent  of  the  Year  3 student  sample. 


7 A small  percentage  of  leavers  moved  to  another  school  in  the  study  and  so  remained  in  the  study.  For  descriptive 
purposes,  they  took  on  the  assignment  of  their  school  to  treatment  or  control.  For  analysis  purposes,  they  retained  their 
initial  assignment.  The  percentages  of  Cohort  1 students  who  switched  status  at  each  round  grew  at  each  follow-up  point 
but  remained  relatively  small  (growing  from  about  0.2%  to  1.4%). 
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Table  1.3.  Characteristics  of  the  student  sample  in  the  multiprogram  evaluation 


Year  1 

Year  1 

Year  2 

Year  3 

(Fall  3rd  grade) 

(Spring  3rd  grade) 

(Spring  4th  grade) 

(Spring  5th  grade) 

Characteristic 

Total 

T reat- 
ment 

Control 

Total 

T reat- 
ment 

Control 

Total 

T reat- 
ment 

Control 

Total 

T reat- 
ment 

Control 

School  sample  size 

84 

42 

42 

84 

42 

42 

83 

42 

41 

83 

42 

41 

Student  sample  size 

6,567 

3,367 

3,200 

6,597 

3,388 

3,209 

6,415 

3,327 

3,088 

6,249 

3,172 

3,077 

Stayers 

t 

t 

t 

6,203 

3,188 

3,015 

5,110 

2,625 

2,485 

4,500 

2,289 

2,211 

New  entrants 

t 

t 

t 

394 

200 

194 

1,305 

702 

603 

1,749 

883 

866 

New  entrants  as  percent  of  spring 
enrollment1 

t 

t 

t 

6.0 

5.9 

6.0 

20.3 

21.1 

19.5 

27.9 

27.8 

28.1 

Total  leavers  (from  original  cohort) 

t 

t 

t 

364 

179 

185 

1,457 

742 

715 

2,067 

1,078 

989 

Leavers  as  percent  of  fall  2004 
enrollment1 

t 

t 

t 

5.5 

5.3 

5.8 

22.2 

22.0 

22.3 

31.5 

32.0 

30.9 

Number  of  students  per  school  (mean) 

78 

80 

76 

79 

81 

76 

77 

80 

76 

75 

76 

75 

Range  of  number  of  students  per  school 

23-149 

38-149 

23-148 

26-155 

35-147 

26-155 

25-151 

36-151 

25-144 

25-164 

28-164 

25-139 

t Not  applicable. 

1 No  statistically  significant  differences  were  found  between  values  for  treatment  and  control  groups. 
SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Because  leavers  were  not  followed,  the  SACD  evaluation  is  not  an  intent -to-treat  impact  analysis  (an  analysis 
that  follows  all  students  originally  assigned  to  the  treatment  or  control  groups).  Therefore,  there  is  the 
possibility  that  the  SACD  programs  caused  differential  student  and  teacher  mobility  in  the  treatment  schools 
versus  mobility  in  the  control  schools.  If  this  occurred,  the  estimates  of  the  impact  of  the  programs  would 
include  both  the  extent  to  which  the  programs  improved  the  outcomes  of  the  average  student  and  the 
mobility  effects  resulting  from  any  differences  in  the  average  outcomes  of  treatment  and  control  students 
who  entered  or  left  the  schools  after  random  assignment  was  completed  in  third  grade.  Descriptive  analyses, 
presented  later,  do  not  identify  differential  mobility  in  the  treatment  and  control  schools,  although  this  cannot 
be  definitively  established. 

Table  1.4  provides  information  on  the  student  sample  by  program.  The  same  trends  in  the  growth  in  the 
percentages  of  leavers  and  new  entrants  were  apparent  but  there  were  differences  among  programs  in  this 
growth.  For  example,  by  Year  3 the  percentages  of  leavers  ranged  from  9 percent  to  50  percent,  and  the 
percentages  of  new  entrants  ranged  from  14  percent  to  38  percent  among  programs.  The  table  also  shows 
that  over  all  3 years  there  was  a statistically  significant  difference  between  the  treatment  and  control  groups 
within  a program  for  the  percentage  of  leavers  in  six  cases  and  the  percentage  of  new  entrants  in  one  case. 

Student  turnover  could  have  affected  the  results  of  the  evaluation.  First,  if  the  SACD  programs  affected 
turnover — and  thus  altered  the  composition  of  the  student  population — then  the  difference  between  the 
treatment  and  control  groups  would  reflect  (1)  the  changes  in  student  behavior  that  were  due  to  the 
intervention,  and  (2)  the  changes  that  arose  indirectly  because  the  student  population  changed.  Second, 
students  had  differing  amounts  of  exposure  to  the  program,  and  this  may  have  affected  the  benefits  to 
students.  In  response  to  these  issues,  sensitivity  analyses  were  conducted  to  address  whether  there  were 
differences  between  the  estimated  impacts  of  the  SACD  programs  for  the  stayers  and  for  the  new  entrant 
subgroups. 
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Table  1.4.  Student  sample  overall  and  by  program  for  all  students  and  for  the  treatment  and  control  groups 


Year  1 

(Fall  3rd  grade) 

Year  1 

(Spring  3rd  grade) 

Year  2 

(Spring  4th  grade) 

Year  3 

(Spring  5th  grade) 

Intervention  program 

All 

students 

All 

students 

Leavers 

New 

entrants 

All 

students 

Leavers 

New 

entrants 

All 

students 

Leavers 

New 

entrants 

# 

V 

# 

# 

V 

# 

To 

# 

V 

# 

All  programs 

6,567 

6,597 

364 

6 

394 

6 

6,415 

1,457 

22 

1,305 

20 

6,249 

2,067 

31 

1,749 

28 

Treatment  group 

3,367 

3,388 

179 

5 

200 

6 

3,327 

742 

22 

702 

21 

3,172 

1,078 

32 

883 

28 

Control  group 

3,200 

3,209 

185 

6 

194 

6 

3,088 

715 

22 

603 

20 

3,077 

989 

31 

866 

28 

ABC 

879 

875 

43 

5 

39 

4 

877 

160 

18 

158 

18 

871 

289 

33 

281 

32 

Treatment  group 

380 

373 

17 

4 

10 

3** 

367 

72 

19 

59 

16 

353 

135 

36 

108 

31 

Control  group 

499 

502 

26 

5 

29 

6 

510 

88 

18 

99 

19 

518 

154 

31 

173 

33 

CSP 

959 

975 

36 

4 

52 

5 

969 

230 

24 

240 

25 

947 

238 

25 

226 

24 

Treatment  group 

476 

485 

20 

4 

29 

6 

474 

135 

28** 

133 

28* 

458 

139 

29** 

121 

26 

Control  group 

483 

490 

16 

3 

23 

5 

495 

95 

20 

107 

22 

489 

99 

20 

105 

21 

LBW 

986 

1,007 

60 

6 

81 

8 

959 

228 

23 

201 

21 

944 

308 

31 

266 

28 

Treatment  group 

548 

565 

25 

5** 

42 

7 

556 

110 

20 

118 

21 

567 

145 

26** 

164 

29 

Control  group 

438 

442 

35 

8 

39 

9 

403 

118 

27 

83 

21 

377 

163 

37 

102 

27 

PA 

811 

812 

74 

9 

75 

9 

764 

251 

31 

204 

27 

655 

408 

50 

252 

38 

Treatment  group 

410 

416 

33 

8 

39 

9 

425 

108 

26** 

123 

29 

327 

209 

51 

126 

39 

Control  group 

401 

396 

41 

10 

36 

9 

339 

143 

36 

81 

24 

328 

199 

50 

126 

38 

PATHS 

786 

783 

39 

5 

36 

5 

778 

150 

19 

142 

18 

778 

243 

31 

235 

30 

Treatment  group 

377 

374 

21 

6 

18 

5 

373 

66 

18 

62 

17 

378 

114 

30 

115 

30 

Control  group 

409 

409 

18 

4 

18 

4 

405 

84 

21 

80 

20 

400 

129 

32 

120 

30 

4Rs 

1,202 

1,194 

86 

7 

78 

7 

1,109 

320 

27 

227 

20 

1,065 

492 

41 

355 

33 

Treatment  group 

652 

647 

49 

8 

44 

7 

599 

183 

28 

130 

22 

556 

279 

43 

183 

33 

Control  group 

550 

547 

37 

7 

34 

6 

510 

137 

25 

97 

19 

509 

213 

39 

172 

34 

See  notes  at  end  of  table. 
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Table  1.4.  Student  sample  overall  and  by  program  for  all  students  and  for  the  treatment  and  control  groups — Continued 


Intervention  program 

Year  1 

(Fall  3rd  grade) 

Year  1 

(Spring  3rd  grade) 

Year  2 

(Spring  4th  grade) 

Year  3 

(Spring  5th  grade) 

All  All 

students  students 

Leavers 

New 

entrants 

All 

students 

Leavers 

New 

entrants 

All 

students 

Leavers 

New 

entrants 

# 

“/o1 

# 

%2 

# 

“/o1 

# 

%2 

# “/o1 

# 

%2 

SS 

944 

951 

26 

3 

33 

3 

959 

118 

13 

133 

14 

989 

89  9 

134 

14 

Treatment  group 

524 

528 

14 

3 

18 

3 

533 

68 

13 

77 

14 

533 

57  11* 

66 

12 

Control  group 

420 

423 

12 

3 

15 

4 

426 

50 

12 

56 

13 

456 

32  8 

68 

15 

* Treatment  group  significantly  different  from  control  group  at  the  .05  level. 

**  Treatment  group  significantly  different  from  control  group  at  the  .01  level. 

1 Leavers  as  a percentage  of  fall  2004  enrollment  (these  values  are  cumulative  over  the  years). 

2 New  entrants  as  a percentage  of  spring  enrollment. 

NOTE:  Abbreviations  are 

ABC:  Academic  and  Behavioral  Competencies  Program 
CSP:  Competence  Support  Program 
LBW:  Love  In  a Big  World 
PA:  Positive  Action 

PATHS:  Promoting  Alternative  Thinking  Strategies 
to  4Rs:  The  4Rs  Program  (Reading,  Writing,  Respect,  and  Resolution) 

1 SS : Second  Step 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Measures 

The  SACD  multiprogram  evaluation  included  six  data  collection  reports:  (a)  Child  Report;  (b)  Primary 
Caregiver  Report;  (c)  Teacher  Report  on  Student;  (d)  Teacher  Report  on  Classroom  and  School;  (e)  Principal 
Interview;  and  (f)  Implementation  Fidelity  Rating.8  The  measures  included  in  these  reports  were  largely  taken 
from  instruments  used  in  previous  research.  Four  measures  (child  responsibility,  child  and  teacher 
perceptions  of  school  safety,  and  primary  caregiver  perceptions  of  community  resources)  were  developed  for 
the  SACD  evaluation  due  to  a lack  of  adequate  existing  measures.  Modifications  were  made  to  some  of  the 
measures  to  enhance  readability,  reduce  the  number  of  items,  or  reduce  the  number  of  possible  responses  to 
each  item.  Children  were  asked  to  respond  based  on  their  experiences  in  the  past  couple  of  weeks;  primary 
caregivers  and  teachers  responded  based  on  their  experiences  in  the  past  30  days.  The  surveys  were  pilot- 
tested  in  December  2003,  and  revisions  were  made  prior  to  the  initial  administration  in  fall  2004.  A series  of 
exploratory  and  confirmatory  factor  analyses  using  the  initial  data  collapsed  the  scales  into  a smaller  number 
of  more  reliable  measures  that  were  used  as  outcome  or  moderator  variables  in  the  impact  analyses.  In 
addition  to  developing  the  outcome  variables,  these  analyses  provided  a check  of  how  well  the  measures  fit 
under  the  outcome  domains  as  set  out  in  the  conceptual  model  (Social  and  Emotional  Competence,  Behavior, 
Academics,  and  Perceptions  of  School  Climate).  The  discussion  of  the  measures  is  broken  into  two  parts. 
First,  the  scales  collected  are  described  by  report  and  domain.  Their  published  reliabilities,  if  available,  are 
reported.9  Second,  the  outcome  and  moderator  measures  developed  through  the  factor  analyses  and  then 
used  in  the  analyses  for  the  SACD  evaluation  are  described. 

Child  Report 

The  Child  Report  was  a group-administered  questionnaire  consisting  of  a battery  of  measures  that  are 
described  below.  The  instrument  was  delivered  to  groups  averaging  15  to  20  students  in  classrooms  at  each 
school  during  a 1-  to  2-day  visit  (or  during  a make-up  session)  by  the  MPR  data  collection  team.  A proctor 
read  the  directions  aloud  as  children  followed  along  in  their  survey  booklets  during  a 50-minute  classroom 
session.  Children  reported  on  their  own  thoughts,  feelings,  and  behaviors.  There  is  some  evidence  in  the 
literature  that  self-reports  of  behavior  problems  are  not  valid  in  elementary-age  aggressive  and  ADHD 
children  (McMahon  and  Frick  2005;  Pelham,  Fabiano,  and  Massetti  2005),  which  may  limit  the  value  of  such 
measures  for  this  type  of  student  (though  similar  measures  were  used  with  primary  caregivers  and  teachers). 
The  following  measures  (grouped  by  domain)  were  included  in  the  battery: 

I.  Social  and  Emotional  Competence 

A.  Beliefs  about  aggression.  This  measure  is  an  8-item  modified  version  of  the  Normative  Beliefs  About 
Aggression  Scale  (Huesmann  and  Guerra  1997).  Children  responded  on  a 4-point  scale, 
indicating  their  beliefs  about  the  acceptability  of  verbal  and  physical  aggression,  either  proactive 
or  in  retaliation  for  others’  behavior  (e.g.,  how  wrong  or  okay  it  is  to  hit,  shove,  fight  with,  or 
verbally  assault  others).  Its  reliability  has  been  evaluated  through  internal  consistency  (0.90). 

B.  Self-efficacy  for  peer  interaction.  This  measure  is  a 12-item  modified  version  of  the  Self-Efficacy  for 
Peer  Interaction  Scale  (Wheeler  and  Ladd  1982).  Children  responded  on  a 4-point  scale, 


8 In  addition,  the  SACD  evaluation  intended  to  include  two  additional  reports  for  each  program:  a cost  report  and  a 
report  on  the  presence  of  artifacts  indicating  the  use  of  SACD  activities.  For  the  cost  report,  a common  training  on 
collection  of  cost  data  was  provided  for  the  research  teams;  however,  all  teams  did  not  collect  such  data  and  the  data 
collected  were  not  consistent.  As  a result,  cost  reports  were  not  generated  and  the  SACD  evaluation  did  not  include  a 
cost  analysis.  For  the  artifact  study,  school  and  classroom  observations  were  done  in  Year  1.  However  inter-rater 
reliability  was  considered  too  low,  so  the  observations  were  not  continued  in  later  years  and  the  Year  1 results  are  not 
reported. 

9 Five  of  the  measures  have  no  published  reliabilities,  and  no  reliabilities  were  computed  for  these  measures  based  on  the 
study  data.  Items  from  these  measures  were  used  to  create  the  outcome  measures.  The  outcome  measures  they 
contributed  to  and  the  reliabilities  of  those  outcome  measures  calculated  from  the  study  data  are  shown  in  table  5. 
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indicating  their  perceived  ability  to  perform  verbal  or  persuasive  prosocial  skills  in  conflict  and 
nonconflict  peer  interactions  (e.g.,  how  hard  or  easy  it  is  to  engage  in  play  with  other  children  or 
resist  negative  social  pressures).  Its  reliability  has  been  evaluated  through  internal  consistency 
(0.85)  and  test-retest  (0.90  for  boys  and  0.80  for  girls). 

C.  Empathy.  This  measure  is  a 16-item  modified  version  of  the  Children’s  Empathy  Questionnaire 
(Funk  et  al.  2003).  Children  responded  on  a 3-point  scale  about  the  degree  to  which  they  would 
respond  empathically  to  hypothetical  and  actual  anticipated  events  (e.g.,  whether  other  people’s 
problems  bother  them  or  if  they  would  feel  happy  when  a friend  gets  a good  grade).  Its  reliability 
has  been  evaluated  through  internal  consistency  (0.72). 

II.  Behavior 

A.  Altruistic  behavior.  The  Altruism  Scale,  Child  Version  (Soloman  et  al.  2000)  is  an  8-item  measure  to 
which  children  responded  on  a 4-point  scale  about  how  often  they  engage  in  helping  behavior 
(e.g.,  cheer  someone  up  who  is  feeling  sad  or  help  someone  who  is  being  picked  on).  Its 
reliability  has  been  evaluated  through  internal  consistency  (0.82). 

B.  Aggression.  The  Aggression  Scale  (Orpinas  and  Frankowski  2001)  is  a 6-item  measure  to  which 
children  responded  on  a 4-point  scale  about  how  often  they  engage  in  verbal  or  physical 
aggression  (e.g.,  tease  others,  shove  others).  Its  reliability  has  been  evaluated  through  internal 
consistency  (0.87). 

C.  Minor  delinquent).  The  modified  Frequency  of  Delinquent  Behavior  (Dahlberg,  Toal,  and  Behrens 
1998;  Dunford  and  Elliott  1984)  Scale  was  modified  for  the  SACD  multiprogram  evaluation. 
Children  responded  to  6 items  on  a 4-point  scale  about  how  often  they  engage  in  minor 
delinquency  in  school  (e.g.,  taking  something  that  belongs  to  others,  skipping  class).  There  is  no 
published  or  study-based  reliability  for  this  scale. 

III.  Academics 

A.  School  engagement.  The  Engagement  vs.  Disaffection  with  Learning  Scale,  (Furrer  and  Skinner 
2003)  is  a 1 0-item  measure  to  which  children  responded  on  a 4-point  scale,  indicating  how  much 
they  agree  with  statements  indicative  of  their  behavioral  and  emotional  engagement,  effort, 
attention,  and  persistence  in  the  classroom  (e.g.,  paying  attention  in  class,  listening  carefully, 
joining  in  class  discussion).  Its  reliability  has  been  evaluated  through  internal  consistency  (0.75  to 
0.86). 

B.  Standardised  test  scores  and  grades.  The  research  teams  were  to  collect  student  standardized  test 
scores  and  grades.  However,  not  all  teams  were  able  to  collect  these  data  from  their  schools  and 
districts,  and  the  data  that  were  collected  varied  in  quality.  As  a result,  neither  standardized  test 
scores  nor  grades  could  be  used  as  outcome  measures  in  the  evaluation,  although  they  had  been 
proposed  for  such  use  in  the  conceptual  model. 

IV.  Perceptions  of  School  Climate 

A.  School  connectedness.  The  Sense  of  School  as  a Community  Scale  (Roberts,  Horn,  and  Battistich 
1995)  is  a 14-item  measure  to  which  children  responded  on  a 4-point  scale  about  how  much  they 
agree  with  statements  indicative  of  the  social  relationships  among  students,  teachers,  and  staff 
within  a school  (e.g.,  students  care  about  each  other,  students  and  teachers  treat  each  other  with 
respect).  Its  reliability  has  been  evaluated  through  internal  consistency  (0.91). 

B.  Safety.  The  Feelings  of  Safety  at  School  Scale  was  developed  for  the  SACD  multiprogram 
evaluation  and  is  a 5-item  measure  to  which  children  responded  on  a 4-point  scale  about  how 
much  they  agree  with  statements  that  are  indicative  of  students’  feelings  of  safety  at  school  (e.g., 
students  feel  safe,  students  are  afraid  someone  will  hurt  them,  students  are  afraid  someone  will 
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tease  them).  The  scale  was  constructed  by  IES  and  DVP  staff  for  the  SACD  multiprogram 
evaluation.  There  is  no  published  or  study-based  reliability  for  this  scale. 

C.  1 Victimisation.  The  Victimization  Scale  (Orpinas  and  Kelder  1995)  is  a 6-item  measure  to  which 
children  responded  on  a 4-point  scale  about  how  often  they  are  victimized  at  school  (e.g.,  teased, 
pushed,  or  threatened).  Its  reliability  has  been  evaluated  through  internal  consistency  (0.90). 

Primary  Caregiver  Report 

The  Primary  Caregiver  Report  was  a self- administered  survey  that  took  primary  caregivers  approximately  15 
minutes  to  complete.  Primary  caregivers,  who  were  usually  but  not  always  parents,  reported  on  the  target 
child’s  behavior,  demographic  factors,  and  home  and  community  environment.  Primary  caregivers  completed 
the  questionnaire  at  home  and  returned  it  to  school  in  a sealed  envelope.  If  a completed  Primary  Caregiver 
Report  was  not  returned  within  3 or  4 weeks,  MPR  attempted  to  complete  a computer-assisted  telephone 
interview  with  the  primary  caregiver.  Primary  caregivers  were  offered  a small  financial  incentive  as  a thank- 
you.  The  following  measures  (grouped  by  domain  or  type  of  moderator)  were  used: 

I.  Behavior 

A.  F 'responsibility . The  Responsibility  Scale  is  an  8-item  measure  developed  for  the  SACD 
multiprogram  evaluation  to  which  primary  caregivers  responded  on  a 4-point  scale  about  the 
degree  to  which  a child  takes  responsibility  for  his  or  her  own  actions  (e.g.,  how  often  a child 
asks  permission,  apologizes,  takes  care  of  property,  or  denies  responsibility  for  wrongdoing).  The 
scale  was  constructed  by  IES  and  DVP  staff  for  the  SACD  multiprogram  evaluation.  There  is  no 
published  or  study-based  reliability  for  this  scale. 

B.  Self -regulation,  cooperation,  and  prosocial  behavior.  The  Social  Competence  Scale  (CPPRG  1999)  is  a 19- 
item  measure  to  which  primary  caregivers  responded  on  a 4-point  scale  about  a child’s  emotional 
and  behavioral  regulation,  cooperative  behavior,  and  prosocial  behavior  (e.g.,  how  often  a child 
calms  down  when  wound  up,  works  well  in  groups,  and  acts  friendly  toward  others).  Its 
reliability  has  been  evaluated  through  internal  consistency  (0.87). 

C.  Altruistic  behavior.  The  Altruism  Scale,  Primary  Caregiver  Version  (Soloman  et  al.  2000)  is  an  8- 
item  measure  to  which  primary  caregivers  responded  on  a 4-point  scale  about  how  often  their 
child  engages  in  helping  behavior  (e.g.,  cheers  someone  up  who  is  feeling  sad  or  helps  someone 
who  is  being  picked  on).  Its  reliability  has  been  evaluated  through  internal  consistency  (0.82). 

D.  Aggression.  The  Behavior  Assessment  System  for  Children  (BASC)  Aggression  Subscale,  Parent 
Version  (Reynolds  and  Kamphaus  1998)  is  a 13-item  assessment  to  which  primary  caregivers 
responded  on  a 4-point  scale  about  how  often  their  child  engages  in  verbal  or  physical  aggression 
(e.g.,  teases  others,  hits  other  children).  Its  reliability  has  been  evaluated  through  internal 
consistency  (0.83)  and  test-retest  (0.84). 

E.  Conduct  problems.  The  BASC  Conduct  Problems  Subscale,  Parent  Version  (Reynolds  and 
Kamphaus  1998)  is  an  11 -item  assessment  to  which  primary  caregivers  responded  on  a 4-point 
scale  about  how  often  their  child  engages  in  socially  deviant  and  disruptive  behaviors  (e.g.,  lies, 
gets  into  trouble).  Its  reliability  has  been  evaluated  through  internal  consistency  (0.82)  and  test- 
retest  (0.92). 

II.  Perceptions  of  School  Climate 

A.  Parent  involvement.  The  modified  Parent  and  Teacher  Involvement  Questionnaire,  Parent  Version 
(CPPRG  1991)  is  a 9-item  measure  to  which  primary  caregivers  responded  on  a 4-point  scale 
about  how  often  they  interact  with  teachers  and  school  staff,  participate  in  school  activities,  and 
are  involved  in  their  children’s  academic  lives  (e.g.,  call  the  child’s  teacher,  attend  parent-teacher 
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conferences,  take  the  child  to  the  library).  Its  reliability  has  been  evaluated  through  internal 
consistency  (0.77). 

III.  Family  Moderators 

A.  Parenting  practices.  The  Alabama  Parenting  Questionnaire,  Positive  Parenting,  and  Poor 
Monitoring  and  Supervision  Subscales  (Shelton,  Frick,  and  Wootton  1996)  is  a 16-item  measure 
to  which  primary  caregivers  responded  on  a 4-point  scale  about  how  often  they  support  and 
reward  their  child,  and  monitor  and  supervise  their  child  (e.g.,  praise  the  child  for  good  behavior, 
check  when  the  child  comes  home).  Its  reliability  has  been  evaluated  through  internal 
consistency  (0.75  for  monitoring  and  0.85  for  positive  parenting). 

B.  Home  atmosphere.  The  modified  Confusion,  Hubbub,  and  Order  Scale  (Matheny  et  al.  1995)  is  a 
14-item  measure  to  which  primary  caregivers  responded  on  a 5-point  scale  about  how  much  they 
agree  or  disagree  with  statements  that  reflect  disorganization  in  the  home  (e.g.,  can  find  things 
when  you  need  them,  there  is  always  a fuss  going  on,  there  is  a regular  routine).  Its  reliability  has 
been  evaluated  through  internal  consistency  (0.79)  and  through  test-retest  (0.74). 

C.  Socioeconomic  risk.  Primary  caregivers  completed  a series  of  questions  indicating  ethnicity, 
education,  employment  status,  household  composition,  and  income. 

IV.  Community  Moderators 

A.  Perceptions  of  community  risks  and  protective  factors.  The  Community  Risks  and  Protective  Factors  Scale 
is  a 12-item  measure  developed  for  the  multiprogram  evaluation  based  on  research  on 
community  risks  by  Forehand,  Brody,  and  Armistead  (2000)  and  supplemented  with  items  to 
assess  community  resources  constructed  by  researchers  involved  in  the  SACD  evaluation. 
Primary  caregivers  indicated  on  a 4-point  scale  how  much  statements  about  community 
characteristics  described  the  neighborhood  the  primary  caregiver  and  child  live  in  (e.g.,  trash 
pickup  is  a problem  in  the  neighborhood,  there  are  safe  outdoor  parks  for  children  to  play  in). 
Seven  of  the  items  were  taken  from  the  original  Community  Risks  Scale  whose  reliability  has 
been  evaluated  through  internal  consistency  (0.89).  The  other  five  items  were  from  the 
Community  Resources  Scale  constructed  by  IES  and  DVP  staff  for  the  SACD  multiprogram 
evaluation  and  for  which  there  is  no  published  or  study-based  reliability. 

B.  Intergenerational  closure.  The  Intergenerational  Closure  Scale  (Sampson,  Morenoff,  and  Earls  1999) 
is  a 5-item  measure  to  which  primary  caregivers  responded  on  a 4-point  scale  to  statements 
about  adults’  connections  with  children  in  the  neighborhood  (e.g.,  parents  know  their  children’s 
friends,  there  are  adults  for  kids  to  look  up  to).  Its  reliability  has  been  evaluated  through  internal 
consistency  (0.72). 

C.  Child-centered  social  control.  The  Child-Centered  Social  Control  Scale  (Sampson,  Morenoff,  and 
Earls  1999)  is  a 5-item  measure  on  which  primary  caregivers  indicated  on  a 5-point  scale  how 
likely  it  is  that  neighbors  could  be  counted  on  to  “do  something”  if  something  occurred  in  the 
neighborhood  (e.g.,  youth  sprayed  graffiti,  children  were  hurt,  neighbors  were  in  need).  Its 
reliability  has  been  evaluated  through  internal  consistency  (0.74). 

Teacher  Report  on  Student 

The  Teacher  Report  on  Student  was  a self-administered  survey  completed  by  teachers  of  the  student  cohort 
in  the  study  (i.e.,  third-grade  teachers  in  2004-05,  fourth-grade  teachers  in  2005-06,  and  fifth-grade  teachers  in 
2006-07).  Teachers  reported  on  each  consented  student’s  typical  behavior  in  the  past  30  days.  Teachers 
completed  the  questionnaires  independently  and  returned  the  surveys  to  a central  point  in  the  school  for 
collection  by  the  research  team.  The  survey  took  approximately  1 5 minutes  for  teachers  to  complete  for  each 


31 


Chapter  1 . The  Social  and  Character  Development  Multiprogram  Evaluation 


consented  student  in  their  classroom.  Teachers  were  offered  a small  financial  incentive  as  a thank-you.  The 
following  measures  (grouped  by  domain)  were  used: 

I.  Behavior 

A.  'Responsibility . The  Responsibility  Scale  is  an  8-item  measure  developed  for  the  SACD 
multiprogram  evaluation.  It  uses  teachers’  responses  on  a 4-point  scale  to  measure  the  degree  to 
which  a child  takes  responsibility  for  his  or  her  own  actions  (e.g.,  how  often  a child  asks 
permission,  apologizes,  takes  care  of  property,  or  denies  responsibility  for  wrongdoing)  for  the 
SACD  multiprogram  evaluation.  The  scale  was  constructed  by  IES  and  DVP  staff  for  the  SACD 
multiprogram  evaluation.  There  is  no  published  or  study-based  reliability  for  this  scale. 

B.  Self -regulation,  cooperation,  and  prosocial  behavior.  The  Social  Competence  Scale  (CPPRG  1999)  is  an 
19-item  measure  to  which  teachers  responded  on  a 4-point  scale  about  a child’s  emotional  and 
behavioral  regulation,  cooperative  behavior,  and  prosocial  behavior  (e.g.,  how  often  a child  calms 
down  when  wound  up,  works  well  in  groups,  and  acts  friendly  toward  others).  Its  reliability  has 
been  evaluated  through  internal  consistency  (0.87). 

C.  Altruistic  behavior.  The  Altruism  Scale  (Soloman  et  al.  2000)  is  an  8-item  measure  to  which 
teachers  responded  on  a 4-point  scale  about  how  often  a student  engages  in  helping  behavior 
(e.g.,  cheers  someone  up  who  is  feeling  sad  or  helps  someone  who  is  being  picked  on).  Its 
reliability  has  been  evaluated  through  internal  consistency  (0.82). 

D.  Aggression.  The  BASC  Aggression  Subscale,  Teacher  Version  (Reynolds  and  Kamphaus  1998)  is  a 
14-item  assessment  to  which  teachers  responded  on  a 4-point  scale  about  how  often  a student 
engages  in  verbal  or  physical  aggression  (e.g.,  bullies  others,  calls  other  children  names).  Its 
reliability  has  been  evaluated  through  internal  consistency  (0.95)  and  through  test-retest  (0.91). 

E.  Conduct  problems.  The  BASC  Conduct  Problems  Subscale,  Teacher  Version  (Reynolds  and 
Kamphaus  1998)  is  a 10-item  assessment  to  which  teachers  responded  on  a 4-point  scale  about 
how  often  a student  engages  in  socially  deviant  or  disruptive  behavior  in  school  (e.g.,  skips  class, 
cheats,  steals).  Its  reliability  has  been  evaluated  through  internal  consistency  (0.77)  and  through 
test-retest  (0.80). 

F.  Attention  deficit  hyperactivity  disorder  (ADHD) -related  behavior.  This  measure  is  a set  of  1 0 items  that 
assess  a range  of  symptoms  of  attention  deficits  and  hyperactivity  such  as  inattention, 
distractibility,  verbal  and  physical  impulsivity,  losing  things,  and  difficulty  organizing  activities.  It 
draws  on  two  sources.  Five  inattention/ overactivity  items  were  taken  from  the  IOWA  Conners 
Teacher  Rating  Scale  (Loney  and  Milich  1982).  The  reliability  of  the  original  Conners  scale  has 
been  evaluated  through  internal  consistency  (0.89  to  0.92).  Five  items  were  based  on  diagnostic 
criteria  for  ADHD  from  the  Diagnostic  and  Statistical  Manual  for  Mental  Disorders,  fourth  edition 
(American  Psychiatric  Association  2000)  that  have  been  shown  to  be  among  the  most  powerful 
for  predicting  ADHD  diagnoses  in  school  settings  (Pelham  et  al.  1992). 

II.  Academics 

A.  Academic  competence.  The  Academic  Competence  and  Motivation  Scale  was  developed  for  the 
SACD  multiprogram  evaluation  and  is  based  on  the  academic  competency  subscales  of  the 
Social  Skills  Rating  System  (Gresham  and  Elliott  1990)  and  items  from  the  Teacher’s  Report 
Form  (Achenbach  1991).  On  this  5-item  measure,  teachers  indicated  how  a student  performed 
overall,  performed  in  reading,  performed  in  math,  and  functioned  intellectually  compared  to 
grade-level  expectations;  and  indicated  how  motivated  a student  was  compared  to  the  average 
student  at  grade  level.  The  reliability  of  the  Social  Skills  Rating  System  has  been  evaluated 
through  internal  consistency  (0.96)  and  through  test-retest  (0.93).  The  reliability  of  the  Teacher 
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Report  Form  has  been  evaluated  through  internal  consistency  (0.72  to  0.95)  and  through  test- 
retest  (0.62  to  0.96). 

III.  Perceptions  of  School  Climate 

A.  Parent  and  teacher  involvement.  This  measure  is  a 7-item  modified  version  of  the  Parent  and  Teacher 
Involvement  Questionnaire,  Teacher  Version  (CPPRG  1991).  Teachers  responded  on  a 4-point 
scale  indicating  how  often  a student’s  primary  caregiver  interacts  with  the  teachers  at  school  and 
attends  school  events  (e.g.,  calls  on  the  phone,  attends  parent-teacher  conferences).  Teachers 
also  responded  to  three  questions  about  communication  with  a student’s  parents,  the  degree  of 
involvement  a parent  has  in  a student’s  life,  and  the  academic  encouragement  provided  to  a 
student.  Its  reliability  has  been  evaluated  through  internal  consistency  (0.77). 

Teacher  Report  on  Classroom  and  School 

The  Teacher  Report  on  Classroom  and  School  was  a self- administered  questionnaire  that  took  a teacher 
approximately  30  minutes  to  complete.  Teachers  reported  on  the  organizational  climates  of  their  schools,  the 
social  and  character  development  activities  they  had  implemented  in  their  classrooms,  and  their  own 
professional  backgrounds.  The  questionnaire  was  distributed  to  third-,  fourth-,  and  fifth-grade  teachers  at 
each  school.  The  following  measures  (grouped  by  domain  or  type  of  moderator)  were  used: 

I.  Perceptions  of  School  Climate 

A.  Organisational  climate.  The  School-Level  Environment  Questionnaire  (Rentoul  and  Fraser  1983)  is 
a 56-item  measure  to  which  teachers  responded  on  a 5-point  scale  about  the  degree  to  which 
teachers  are  supported  and  valued  by  administration  officials,  are  provided  adequate  resources, 
are  autonomous  in  managing  their  own  activities,  are  collegial  with  one  another,  and  have  a voice 
in  school  policy  decisions.  Its  reliability  has  been  evaluated  through  internal  consistency  (0.82). 

B.  Safety.  The  Feelings  of  Safety  at  School  Scale  was  developed  for  the  SACD  multiprogram 
evaluation  and  is  a 4-item  measure.  Teachers  responded  on  a 4-point  scale  about  how  much  they 
agree  with  statements  that  are  indicative  of  students’  feelings  of  safety  at  school  (e.g.,  students 
feel  safe  at  this  school,  students  are  afraid  someone  will  hurt  them  at  school,  students  are  afraid 
someone  will  tease  them  at  school).  This  scale  was  constructed  by  IES  and  DVP  staff  for  the 
SACD  multiprogram  evaluation.  There  is  no  published  or  study-based  reliability  for  this  scale. 

II.  Level  of  SACD  in  the  School 

The  SACD  Activities  Teacher  Survey  is  a self-report  questionnaire  constructed  by  SACD  Research 
Consortium  members.  Teachers  indicated  the  SACD-related  programs  and  activities  occurring  in  the 
classroom  and  school,  and  the  degree  to  which  teachers  and  staff  incorporated  SACD-like  qualities  in 
their  professional  relationships  and  styles.  The  questionnaire  consisted  of  11  major  questions,  each 
with  multiple  parts.  A set  of  83  SACD  activities  variables  was  created  from  teacher  responses  to 
individual  questions.  These  variables  were  used  to  describe  the  level  of  social  and  character 
development  formally  occurring  in  every  school.  As  such,  they  were  also  used  as  outcome  variables 
(no  additional  construction  was  done  for  their  use  in  the  analysis)  to  evaluate  the  impact  of  the 
SACD  programs  on  the  use  of  SACD  activities  in  the  schools.  The  individual  outcomes  were 
grouped  under  six  domains:10  (1)  use  of  SACD  activities,  (2)  use  of  SACD  activities  associated  with  a 
named  SACD  program,  (3)  use  of  classroom  materials  and  strategies  to  promote  social  and  character 
development,  (4)  use  of  schoolwide  strategies  to  promote  social  and  character  development,  (5) 
professional  development  of  teachers  to  promote  social  and  character  development,  and  (6)  teachers’ 
attitudes  toward  the  promotion  of  social  and  character  development  and  school  practices  conducive 


10  Outcomes  were  grouped  into  the  six  domains  based  on  face  validity  alone.  No  psychometric  procedures  were  used, 
and  no  measures  of  reliability  or  validity  were  obtained. 
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to  such  promotion.  Below  is  a description  of  each  of  the  SACD  Activities’  variables  developed  from 
the  survey  and  grouped  by  domain. 

A.  Use  of  Social  and  Character  Development  Activities.  Teachers  reported  whether  they  engaged  in 

activities  to  promote  six  SACD  goals  and  behavior  management.  The  six  SACD  goals  were 

defined  as  follows: 

1.  Violence  Prevention  and  Peace  Promotion:  to  make  students  and  schools  safer  and  more 
peaceful  (e.g.,  reducing  verbal  and  physical  aggression,  weapons,  and  bullying  while 
promoting  conflict  resolution  and  peacemaking); 

2.  Social  and  Emotional  Competence:  to  foster  positive  emotional,  behavioral,  and 
interpersonal  skill  (e.g.,  anger  management,  recognizing  emotions,  empathy,  respectful 
communication,  and  building  and  maintaining  friendships); 

3.  Character  Education:  to  encourage  the  development  of  desirable  traits,  values,  and  ethics 
(e.g.,  respect,  responsibility,  honesty,  fairness,  and  caring); 

4.  Tolerance  and  Diversity:  to  promote  understanding  of  and  respect  for  the  ways  in  which 
people  differ  (e.g.,  promote  acceptance  of  others,  celebrate  cultural  and  ethnic  differences, 
and  reduce  prejudice); 

5.  Risk  Prevention  and  Health  Promotion:  to  reduce  unhealthy  behaviors  and  promote  healthy 
ones  (e.g.,  prevent  alcohol,  tobacco,  and  drug  use  and  promote  healthy  life  choices);  and 

6.  Civic  Responsibility  and  Community  Service:  to  promote  participation  in  and  commitment 
to  the  common  good  including  service  to  the  classroom,  school,  and  community  (e.g.,  raising 
resources  for  charity,  recycling,  and  promoting  understanding  of  one’s  role  in  creating 
community). 

For  each  goal,  teachers  were  asked  to  provide  a brief  description  of  any  activity  that  addressed  the 
goal  in  their  class  since  the  beginning  of  the  school  year,  to  indicate  whether  they  did  the  activity  with 
all  or  some  students,  and  the  frequency  and  amount  of  time  devoted  to  these  activities.  Two  sets  of 
seven  variables  were  developed  from  these  responses.  The  first  set  of  seven  variables  was  composed 
of  binary  variables  (yes  or  no)  as  to  whether  teachers  reported  that  they  had  engaged  in  a SACD 
activity  for  each  SACD  goal  separately  (six  variables)  and  whether  they  reported  that  they  had 
engaged  in  a SACD  activity  for  any  of  the  SACD  goals  (one  variable).  The  second  set  of  seven 
variables  was  composed  of  binary  variables  as  to  whether  teachers  reported  that  they  had  engaged  in 
a SACD  activity  for  at  least  1 hour  per  week  for  each  SACD  goal  separately  (six  variables)  and 
whether  they  reported  that  they  had  engaged  in  a SACD  activity  for  at  least  1 hour  per  week  for  any 
of  the  SACD  goals  (one  variable).  One  hour  was  chosen  as  the  cutoff  point  because  most  of  the 
programs  expected  teachers  to  conduct  weekly  lessons  that  would  require  approximately  this  much 
time. 

In  addition,  respondents  also  reported  separately  on  the  use  and  amount  of  time  devoted  to  behavior 
management  activities.  These  reports  were  used  to  create  two  binary  variables:  (1)  whether  teachers 
reported  that  they  had  engaged  in  activities  to  support  behavior  management,  and  (2)  whether 
teachers  reported  that  they  had  engaged  in  activities  to  support  behavior  management  for  at  least  1 
hour  per  week. 

In  total,  this  domain  contains  16  outcome  variables:  (1)  Six  variables  regarding  teachers’  reported  use 
of  SACD  activities  to  promote  each  of  the  six  SACD  goals;  (2)  six  variables  regarding  teachers’ 
reported  use  of  SACD  activities  for  more  than  an  hour  per  week  to  promote  each  of  the  six  SACD 
goals,  (3)  two  variables  regarding  teachers’  reported  promotion  of  any  of  the  six  SACD  goals  using 
SACD  activities  or  using  SACD  activities  for  at  least  an  hour  per  week,  and  (4)  two  variables 
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regarding  teachers’  reported  promotion  of  behavior  management  using  activities  or  using  activities 

for  at  least  an  hour  per  week 

B.  Use  of  Social  and  Character  Development  Activities  Associated  With  a Named  Program.  When  teachers 
provided  the  data  described  above,  regarding  their  engagement  in  activities  to  promote  the  six 
SACD  goals  and  behavior  management,  they  also  could  note  whether  the  activity  was  associated 
with  a specific  SACD  program.  Two  sets  of  7 variables  were  developed  from  these  responses  for 
a total  of  14  variables  within  this  domain.  The  first  set  of  seven  variables  was  composed  of 
binary  variables  (yes  or  no)  as  to  whether  teachers  reported  that  they  had  engaged  in  a SACD 
activity  linked  to  a named  SACD  program  for  each  SACD  goal  separately  (6  variables)  and 
whether  they  reported  that  they  had  engaged  in  a SACD  activity  linked  to  a named  SACD 
program  for  any  of  the  SACD  goals  (1  variable).  The  second  set  of  7 variables  was  composed  of 
binary  variables  as  to  whether  teachers  reported  that  they  had  engaged  in  a SACD  activity  linked 
to  a named  SACD  program  for  at  least  1 hour  per  week  for  each  SACD  goal  separately  (6 
variables)  and  whether  they  reported  that  they  had  engaged  in  a SACD  activity  linked  to  named 
SACD  program  for  at  least  1 hour  per  week  for  any  of  the  SACD  goals  (1  variable).  Few 
respondents  mentioned  a specific  behavior  management  program  by  name,  so  no  measures  were 
created  for  use  of  behavior  management  associated  with  a named  program. 

C.  Classroom  Materials  and  Strategies  Used  to  Promote  Social  and  Character  Development.  Teachers  were 
asked  about  their  use  of  six  specific  types  of  materials  and  20  specific  classroom  strategies  to 
promote  social  and  character  development  in  the  classroom.  Responses  to  these  questions  were 
used  to  create  variables  indicating  the  teacher-reported  use  of  each  type  of  material  and  strategy. 
Twenty-nine  outcome  variables  are  contained  under  this  domain,  including  28  binary  variables 
(yes  or  no)  and  1 count  variable:  (1)  7 binary  variables  regarding  reported  use  of  materials,  with  1 
for  each  of  the  six  types  of  materials  plus  a binary  variable  for  no  report  of  using  any  of  the  six 
materials;  (2)  21  binary  variables  regarding  reported  use  of  the  strategies,  with  1 for  each  of  the 
20  types  of  strategies  plus  a binary  variable  for  reported  use  of  any  of  the  20  strategies;  and  (3)  1 
count  variable  for  the  average  number  of  strategies  reported  used. 

D.  Schoolwide  Strategies  Used  to  Promote  Social  and  Character  Development.  Teachers  were  asked  whether 
their  school  used  six  specific  types  of  schoolwide  strategies  to  promote  social  and  character 
development.  These  included  morning  announcements  or  videos,  school  assemblies,  school 
newspapers  or  bulletins,  special  school  days,  special  events,  or  other  activities.  Responses  to 
these  questions  were  used  to  create  six  binary  variables  for  whether  or  not  the  teacher  reported 
that  the  school  used  each  specific  schoolwide  strategy. 

E.  Professional  Developtnent  of  Teachers  to  Promote  Social  and  Character  Development.  Teachers  were  asked 
about  the  training  they  received  in  instructional  methods  related  to  social  and  character 
development  during  the  prior  12  months.  Information  was  collected  on  whether  any  training  was 
received,  the  total  number  of  hours  of  training,  whether  training  was  received  in  each  of  the  six 
SACD  goal  areas,  and  whether  teachers  participated  in  behavior  management  training.  Nine 
outcome  variables  (eight  binary  and  one  count)  were  developed  from  these  responses:  (1)  one 
binary  variable  as  to  whether  teachers  reported  any  such  training  within  the  past  12  months, 
(2)  seven  binary  variables  as  to  whether  teachers  reported  any  such  training  within  the  past  12 
months  linked  to  each  of  the  six  SACD  goals  separately  or  to  behavior  management,  and  (3)  one 
count  variable  as  to  the  number  of  reported  hours  spent  in  such  training  during  the  past  12 
months. 

F.  Attitudes  Toward  SACD  Efforts  and  School  Practices  Conducive  to  SACD.  Teachers  were  asked  to 
characterize  their  own  attitudes  about  efforts  to  promote  social  and  character  development 
(enthusiastic,  cooperative,  or  in  open  dislike)  and  to  characterize  whether  the  culture  of  their 
school  reflected  the  use  of  six  practices  (most  of  the  time)  conducive  to  the  social  and  character 
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development  of  students.  These  practices  included  teacher  modeling  of  positive  character  and 
behavior  with  students  and  with  other  staff,  involvement  of  students  in  discussions,  students’ 
voice  in  governance,  school  encouragement  of  parent  involvement,  and  discipline  practices  that 
included  promoting  development  rather  than  only  punishing  misbehavior.  Nine  binary  outcome 
variables  were  developed  from  these  responses:  (1)  three  regarding  whether  teachers  reported 
being  enthusiastic,  cooperative,  and  in  open  dislike  to  promoting  social  and  character 
development,  and  (2)  six  regarding  the  reported  use  of  each  school  practice. 

III.  Teacher  Experience 

The  Teacher  Survey  on  Professional  Development  and  Training  was  developed  for  the  SACD 
multiprogram  evaluation  as  a 10-item  questionnaire  to  which  teachers  responded  about  their 
demographic  background,  education,  teaching  experience,  and  engagement  in  professional 
development  activities.  Items  for  this  survey  were  drawn  from  those  used  in  an  earlier  study  on 
teacher  quality  carried  out  by  the  National  Center  for  Education  Statistics  (Lewis  et  al.  1999). 

Principal  Interview 

Principals  were  individually  interviewed  in  a 45-minute  period  about  the  prevalence  and  characteristics  of  the 
activities  used  to  promote  social  and  character  development  and  about  the  percentage  of  staff  trained  in  them 
and  staff  support  of  their  use.  Interviewers  used  both  open-ended  prompts  and  questions  with  defined 
answers.  In  fall  2004  and  spring  2005,  interviews  were  conducted  in  person.  However,  the  recording  of  the 
open-ended  responses  was  not  of  acceptable  quality,  including  differences  in  the  level  of  detail  provided  by 
interviewers.  For  the  second  and  third  years  of  the  project,  principal  interviews  were  done  by  a smaller  group 
of  more  experienced  staff  through  phone  interviews.  Because  of  the  quality  concerns,  the  spring  2005  open- 
ended  data  are  not  reported.  The  principal  interviews  were  used  to  develop  six  measures  of  the  SACD 
implementation  in  all  the  schools. 

I.  Percentages  of  Schools  Using  Activities  to  Address  Each  SACD  Goal 

The  percentages  of  treatment  and  control  schools  for  which  the  principal  noted  that  social  and 
character  development  activities  were  directed  toward  each  of  the  following  six  SACD  goals: 
(1)  Violence  Prevention  and  Peace  Promotion,  (2)  Social  and  Emotional  Development,  (3)  Character 
Education,  (4)  Tolerance  and  Diversity,  (5)  Risk  Prevention  and  Health  Promotion,  and  (6)  Civic 
Responsibility  and  Community  Service.  In  addition,  principals  reported  on  the  number  of  activities 
per  goal.  Both  open-  and  closed-ended  questions  were  used. 

II.  Delivery  Method 

Both  open-  and  closed-ended  questions  were  used  to  code  data  into  yes  or  no  variables  on  1 3 methods 
and  seven  actors  used  to  deliver  activities  to  promote  social  and  character  development.  Delivery 
methods  included  the  use  of  assemblies,  structured  lessons,  targeted  reading  or  writing,  and  words  of 
the  day;  and  the  actors  included  the  classroom  teacher,  other  teachers,  and  guidance  counselors  or 
social  workers. 

III.  Targeting 

Principals  were  asked  to  specify  whether  the  SACD  activities  were  targeted  to  the  whole  school,  specific 
grades,  or  other.  If  the  principal  selected  other,  he  or  she  was  asked  to  describe  which  students  the 
activity  targeted.  Both  open-  and  closed-ended  questions  were  used. 

IV.  Teacher  Professional  Development 

Data  about  teacher  training  were  collected  in  both  open-  and  closed-ended  questions.  In  the  open- 
ended  responses,  principals  described  the  training  teachers  received  for  each  activity  designed  to 
promote  social  and  character  development.  These  responses  were  coded  to  indicate  whether  a 
teacher  received  guided  or  self-guided  training.  In  the  closed-ended  responses,  principals  reported,  at 
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the  school  level,  the  percentages  of  teachers  and  staff  who  received  training  in  activities  to  promote 
social  and  character  development  and  how  many  hours  of  training  teachers  received. 

V.  Staff  Response 

Data  about  the  staff  response  and  support  of  SACD  were  collected  in  the  closed-ended  questions. 
Principals  reported,  at  the  school  level,  on  the  level  of  staff  enthusiasm  for  activities  to  promote 
social  and  character  development.  Principals  also  reported  on  whether  teachers  and  staff  modeled 
positive  behaviors,  whether  students  were  involved  in  decisionmaking  at  the  school,  and  whether 
discipline  strategies  focused  on  promoting  development  rather  than  only  punishing  misbehavior. 

VI.  Parental  Involvement 

Data  on  whether  parents  were  actively  or  passively  involved  in  activities  to  promote  social  and 
character  development  were  coded  from  responses  to  open-ended  questions  that  asked  principals  to 
describe  how  parents  were  involved  in  these  activities. 

Implementation  Fidelity  Rating 

In  order  to  develop  a common  implementation  measure  for  the  multiprogram  evaluation,  the  SACD 
Research  Program  Global  Fidelity'  Ratings  form  was  provided  to  each  grantee  for  each  of  the  3 study  years. 
The  form  contained  a request  that  each  grantee  rate  the  fidelity  with  which  the  SACD  program  was 
implemented  in  each  treatment  school  during  each  school  year.  The  instructions  explained  that,  for  the 
fidelity  ratings,  the  standard  of  comparison  should  be  program  standards  or  benchmarks.  Furthermore,  the 
instructions  explained  that  the  basis  of  all  fidelity  ratings  should  be  the  site-specific  fidelity  data  collected  as 
part  of  the  grantee’s  own  evaluation  work,11  the  program  benchmarks,  and  the  rater’s  own  judgment.  In 
addition  to  rating  each  school,  each  grantee  was  asked  to  provide  a justification  for  each  rating.  Two 
individuals  with  the  most  experience  in  monitoring  program  implementation  in  the  SACD  treatment  schools 
and  interpreting  the  site-specific  fidelity  data  were  to  complete  the  ratings  independently,  discuss  each  rating, 
and  then  decide  on  a final  consensus  rating  for  each  treatment  school. 

For  each  school,  each  rater  was  to  rate  the  amount  of  implementation  and  the  quality  of  implementation. 
After  making  individual  ratings,  each  pair  was  to  discuss  their  ratings  and  arrive  at  a consensus  rating.  The 
rating  values  ranged  from  1 to  5.  For  example,  a rating  of  1 for  the  amount  of  implementation  at  a school 
indicated  that  “On  average,  the  program  was  delivered  at  a level  substantially  below  program  benchmarks.”  A 
rating  of  2 indicated  “On  average,  the  program  was  delivered  at  a level  somewhat  below  program  benchmarks.” 
A rating  of  3 indicated  the  program  met  benchmarks;  while  ratings  of  4 and  5 were  for  delivery  at  levels 
somewhat  above  and  substantially  above  program  benchmarks,  respectively.  The  ratings  for  the  quality  of 
implementation  at  a school  were  analogous.  For  example,  a rating  of  1 for  the  quality  of  implementation  at  a 
school  indicated  that  “On  average,  the  program  was  delivered  with  quality  substantially  below  program 
benchmarks.” 

Construction  of  Outcome  Variables 

To  develop  a set  of  student  and  school  climate  outcome  measures  with  better  psychometric  characteristics 
and  to  determine  whether  a smaller  number  of  scales  could  represent  them,12  a set  of  exploratory  factor 
analyses  using  the  fall  2004  survey  data  for  each  group  of  respondents  (children,  primary  caregivers,  and 


11  For  their  independent  evaluations,  the  research  teams  developed  their  own  approaches  and  used  a variety  of  methods 
(including  training  logs,  teacher  reports,  and  classroom  observations)  to  measure  the  implementation  fidelity  of  their 
separate  programs. 

12  A preliminary  factor  analysis  of  the  fall  2004  data  indicated  that  certain  scales  had  poor  internal  consistency  while 
others  were  highly  intercorrelated. 
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teachers)  was  done  on  a randomly  selected  half  of  the  sample.13  Exploratory  analysis  of  data  for  each  reporter 
found  that  the  item  responses  of  children,  primary  caregivers,  and  teachers  were  optimally  represented  by  10, 
3,  and  5 underlying  factors,  respectively,  for  a total  of  18  constructs.  Factors  representing  2 constructs 
(Altruistic  Behavior  and  Problem  Behavior)  were  identified  for  all  three  respondent  groups.  A factor 
representing  a third  construct  (Positive  Social  Behavior)  was  identified  for  both  primary  caregivers  and 
teachers,  and  a factor  representing  2 highly  related  constructs  was  identified  for  teachers  and  children 
(Academic  Competence  and  Motivation,  and  Engagement  with  Learning).  The  remaining  identified  factors 
were  specific  to  each  responding  group.  These  18  factors  were  subjected  to  a confirmatory  factor  analysis 
with  the  remaining  half  of  the  fall  2004  sample  using  structural  equation  modeling  procedures.  Two 
additional  constructs  (Feelings  of  Safety,  and  Student  Support  for  Teachers)  were  developed  through  a 
principal  component  analysis  of  the  Teacher  Report  on  Classroom  and  School.  Further  analysis  determined 
that  the  results  were  robust  across  gender  and  race/ethnic  subgroups,  individually  for  each  of  the  seven 
SACD  programs,  when  missing  item  responses  were  imputed,  and  when  the  full  fall  2004  and  spring  2005 
data  were  used.  As  a result  of  these  analyses,  20  constructs  were  identified  and  these  were  used  as  the 
outcome  measures  for  the  SACD  evaluation.  Table  1.5  lists  the  outcome  measures  by  report  and  details  the 
number  of  items  in  each,  their  reliability,  and  the  original  measures  that  provided  the  items  for  them.  Changes 
in  these  measures  may  be  beneficial  or  detrimental  depending  on  the  specific  outcome,  so  a positive  change 
does  not  automatically  reflect  an  improvement  or  a negative  change  a decline.  The  +/-  sign  in  parentheses 
next  to  each  outcome  in  the  first  column  of  table  1.6  notes  whether  a gain  or  loss  can  be  interpreted  as 
beneficial  for  each  outcome  (as  do  the  first  columns  of  all  tables  reporting  impacts  on  the  outcomes).  The  20 
outcome  measures  were  grouped  under  four  domains  based  on  the  conceptual  model:  Social  and  Emotional 
Competence,  Behavior,  Academics,  and  Perceptions  of  School  Climate  (table  1.6). 14 


13  See  appendix  B for  details  on  the  development  of  the  outcome  measures. 

14  One  further  analysis  was  done  to  determine  whether  the  multiple  reports  for  each  of  three  constructs  within  the 
Behavior  domain  could  be  combined:  Altruistic  Behavior  (child,  caregiver,  and  teacher),  Positive  Social  Behavior 
(caregiver  and  teacher),  and  Negative  Behavior  (child,  caregiver,  and  teacher).  A series  of  exploratory  multitrait, 
multirespondent  analyses  found  that  these  outcomes  were  affected  by  both  the  behavior  construct  being  assessed  and  by 
the  reporter.  As  a result,  separate  outcome  measures  for  each  reporter  (child,  caregiver,  and  teacher)  were  retained. 
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Report/Outcome  variable 

Items 

Range 

Reliability1 

Original  measure  from  which  items  were  taken 

Child  Report 

Altruistic  Behavior 

8 

0-3 

0.88 

Altruism  Scale,  Child  Version 

Engagement  with  Learning 

4 

1-4 

0.84 

4 items  from  Engagement  vs.  Disaffection  with  Learning  Scale 

Normative  Beliefs  About 

8 

1-4 

0.83 

Normative  Beliefs  About  Aggression  Scale 

Aggression 

Self-Efficacy  for  Peer  Interaction 

12 

1-4 

0.83 

Self-Efficacy  for  Peer  Interaction  Scale 

Empathy 

11 

1-3 

0.78 

1 1 items  from  the  Children’s  Empathy  Questionnaire 

Problem  Behavior 

12 

0-3 

0.86 

6 items  from  Frequency  of  Delinquent  Behavior  Scale  and  6 items  from 
Aggression  Scale 

Positive  School  Orientation 

10 

1-4 

0.86 

9 items  from  Sense  of  School  as  a Community  Scale  and  1 item  from  Feelings  of 
Safety  at  School  Scale 

Negative  School  Orientation 

8 

1-4 

0.78 

4 items  from  Engagement  vs.  Disaffection  with  Learning  Scale  and  4 items  from 
Sense  of  School  as  a Community  Scale 

Student  Afraid  at  School 

4 

1-4 

0.79 

4 items  from  Feelings  of  Safety  at  School  Scale 

Victimization  at  School 

6 

0-3 

0.86 

Victimization  Scale 

Primary  Caregiver  Report 

Altruistic  Behavior 

8 

1-4 

0.88 

Altruism  Scale,  Primary  Caregiver  Version 

Positive  Social  Behavior 

25 

1-4 

0.93 

6 items  from  Responsibility  Scale  and  19  items  from  Social  Competence  Scale 

Problem  Behavior 

20 

1-4 

0.86 

12  items  from  BASC  Aggression  Subscale,  6 items  from  BASC  Conduct 
Problems  Subscale,  and  2 items  from  the  Responsibility  Scale 

See  notes  at  end  of  table. 
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Table  1.5.  Outcome  variables — Continued 


Report/Outcome  variable 

Items 

Range 

Reliability1 

Original  measure  from  which  items  taken 

Teacher  Report  on  Student 

Altruistic  Behavior 

8 

1-4 

0.89 

Altruism  Scale,  Teacher  Version 

Academic  Competence  and 

5 

1-5 

0.95 

Academic  Competence  and  Motivation  Scale 

Motivation 

Positive  Social  Behavior 

25 

1-4 

0.97 

6 items  from  Responsibility  Scale  and  19  items  from  the  Social  Competence 
Scale 

Problem  Behavior 

23 

1-4 

0.95 

14  items  from  BASC  Aggression  Subscale,  Teacher  Version,  7 items  from  BASC 
Conduct  Problems  Subscale,  Teacher  Version  and  2 items  from  Responsibility 
Scale 

ADHD-Related  Behavior 

10 

1-4 

0.91 

5 items  from  DSM-IV  Criteria  for  ADHD  and  5 items  from  IOWA  Conners 
Teacher  Rating  Scale 

Teacher  Report  on  Classroom 
and  School 

Feelings  of  Safety 

4 

1-5 

0.89 

Feelings  of  Safety  at  School  Scale 

Student  Support  for  Teachers 

7 

1-5 

0.89 

7 items  from  the  School-Level  Environment  Questionnaire 

1 Internal  consistency  as  measured  by  Cronbach’s  alpha. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Table  1.6.  Outcomes,  by  domain  and  data  instrument 


Domain/Outcome 

Instrument 

Child  Report 

Primary 

Caregiver 

Report 

Teacher 
Report  on 
Student 

Teacher 
Report  on 
Classroom 
and  School 

Social  and  Emotional  Competence  Domain 

Self-Efficacy  for  Peer  Interaction  (+) 

■/ 

Normative  Beliefs  About  Aggression  (-) 

■/ 

Empathy  (+) 

V 

Behavior  Domain 

Altruistic  Behavior  (+) 

A 

✓ 

if 

Positive  Social  Behavior  (+) 

A 

Problem  Behavior  (-) 

•/ 

ADHD-Related  Behavior  (-) 

■/ 

Academics  Domain 

Engagement  with  Learning  (+) 

V 

Academic  Competence  and  Motivation  (+) 

A 

Perceptions  of  School  Climate  Domain 

Positive  School  Orientation  (+) 

A 

Negative  School  Orientation  (-) 

A 

Student  Afraid  at  School  (-) 

A 

Victimization  at  School  (-) 

A 

Feelings  of  Safety  (+) 

% 

Student  Support  for  Teachers  (+) 

% 

NOTE:  Abbreviations  are 


ADHD:  Attention  deficit  hyperactivity  disorder 

v : Outcome  addressed 

Blank  cell:  Outcome  not  addressed 

The  +/-  signs  in  parentheses  indicate  the  direction  of  a beneficial  outcome. 
SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 


Construction  of  Moderator  Variables 

The  impact  of  the  SACD  programs  was  hypothesized  to  vary  by  student,  family,  and  community 
characteristics  and  by  the  level  of  implementation.  Collected  student  demographic  characteristics  (such  as 
gender)  could  be  directly  used  in  the  analysis.  Other  characteristics  required  the  use  of  scales  and  the 
construction  of  moderator  variables  for  use  in  the  analysis. 

Initial  Risk  Factors 

Certain  individual,  family,  and  community  characteristics  are  associated  with  greater  prevalence  of  child 
behavior  problems.  Students  with  these  characteristics  began  the  SACD  study  with  initial  risks  that  were 
hypothesized  to  lead  to  more  negative  outcomes.  To  help  determine  the  impact  of  these  initial  risks  on  the 
effects  of  the  SACD  programs,  four  risk  factors  were  constructed  to  represent  (1)  socioeconomic  risk, 
(2)  family  risk,  (3)  perceived  community  risk,  and  (4)  child  behavior  risk.  To  select  the  measures  to  be 
included  in  the  first  three  risk  domains,  associations  were  examined  between  the  risk  measures  and  a set  of 
baseline  outcome  measures. 

Socioeconomic  risk  was  defined  as  the  number  of  the  following  risk  factors  present  in  the  child’s  life  at  the 
initial  data  collection:  (1)  the  child  was  in  a single-parent  household,  (2)  the  child  was  in  a low-income 
household  (below  135%  of  the  poverty  level),  and  (3)  the  child’s  primary  caregiver  had  failed  to  graduate 
from  high  school.  A cumulative  risk  index  across  the  three  risk  factors  was  calculated  by  adding  the  number 
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of  these  factors  each  child  had:  0 (no  risk  factors),  1 (one  risk  factor),  and  2 (two  or  three  risk  factors).  Scores 
ranged  from  0 to  2,  with  a mean  of  0.73  and  a standard  deviation  of  0.74.  The  fourth  measure  of 
socioeconomic  risk  collected,  employment  status  of  the  child’s  caregiver,  was  not  included  in  the 
socioeconomic  risk  measure  because  it  was  not  found  predictive  of  baseline  outcomes  when  the  other  three 
measures  were  included. 

Family  risk  was  measured  using  the  Poor  Monitoring  and  Supervision  Subscale  from  the  Alabama  Parenting 
Questionnaire  (Shelton  et  al.  1996)  collected  in  the  Primary  Caregiver  Report.  Scores  ranged  from  1 to  3.5, 
with  a mean  of  1.1  and  standard  deviation  of  0.20.  The  other  two  measures  of  family  risk  collected,  the 
Positive  Parenting  Subscale  and  the  Confusion,  Hubbub,  and  Order  Scale,  appeared  to  reflect  reporter 
variance  rather  than  construct  variance  and  so  were  not  included  in  family  risk  measure. 

Perceived  community  risk  was  measured  using  items  on  the  Community  Risk  Scale  (Forehand  et  al.  2000), 
collected  in  the  Primary  Caregiver  Report.  Scores  ranged  from  1 to  4,  with  a mean  of  1.5  and  a standard 
deviation  of  0.64.  The  other  two  measures  of  community  risk  collected,  the  Community  Resources  Scale  and 
the  Child-Centered  Social  Control  Scale  were  less  consistently  related  to  the  outcome  variables  and  so  were 
not  included  in  the  community  risk  measure. 

Child  behavior  risk  was  measured  using  the  Conduct  Problem  Subscales  from  the  BASC  (Reynolds  and 
Kamphaus  1998)  collected  in  the  Teacher  Report  on  Student  and  the  Primary  Caregiver  Report.  For  the 
teacher-reported  child  behavior  risk  variable,  scores  ranged  from  42  to  120,  with  a mean  of  50.82  and  a 
standard  deviation  of  9.78.  For  the  parent-reported  child  behavior  risk  variable,  scores  ranged  from  36  to  1 13, 
with  a mean  of  50.33  and  a standard  deviation  of  9.02. 

Fidelity  of  Implementation 

How  well  an  intervention  was  implemented  was  hypothesized  to  be  associated  with  how  the  SACD  programs 
affected  the  outcomes.  The  constructs  used  for  the  fidelity  data  analysis  were  developed  so  that  the  SACD 
evaluation  would  be  able  to  examine  whether  impacts  on  student  outcomes  were  estimated  to  be  larger  in 
treatment  group  schools  that  achieved  high-fidelity  implementation  of  their  SACD  program  (relative  to  their 
matched  control  group  counterparts)  than  were  impacts  in  treatment  group  schools  that  achieved  lower 
fidelity  implementation  (relative  to  their  control  group  counterparts).  Data  collection  was  designed  in  such  a 
way  that  each  program  was  to  be  assessed  relative  to  the  benchmarks  or  expectations  established  for  the 
specific  program.  Consensus  ratings  on  a quality-of-implementation  5-point  scale  and  on  a quantity-of- 
implementation  5-point  scale  were  to  be  used  to  categorize  the  treatment  schools  as  high  in  implementation 
fidelity  (high-fidelity  subgroup)  or  low  in  implementation  fidelity  (low-fidelity  subgroup)  (see  Measures 
section).  The  ratings  on  quality  and  quantity  of  implementation  were  found  to  be  highly  correlated  for  each 
school  (0.78  in  Year  1,  0.82  in  Year  2,  and  0.83  in  Year  3).  As  a result,  they  were  combined  into  a single  10- 
point  composite  measure  of  fidelity  of  implementation,  which  was  used  to  create  the  high-  and  low-fidelity 
subgroups. 

The  cutoff  point  on  the  consensus  score  used  to  place  the  schools  into  the  two  fidelity  subgroups  was 
established  using  two  criteria.  First,  every  year  a pair  of  raters  independently  rated  each  school  within  a 
specific  program  and  then  agreed  on  a consensus  score  for  each  school.15  The  cutoff  point  was  to  be  set  so 
that  the  individual  ratings  were  to  agree  on  the  placement  for  the  largest  possible  percentage  of  schools.  The 
agreement  between  the  two  raters’  individual  scores  on  their  schools’  placement  was  measured  using  Cohen’s 
kappa,  which  adjusts  the  empirical  match  rate  by  the  expected  number  of  exact  matches  that  would  occur  by 
chance.  Second,  a sufficient  number  of  schools  were  to  be  placed  in  each  subgroup  so  that  the  size  of  each 


15  Over  the  3 years,  the  ratings  were  not  always  consistently  provided  by  the  research  teams.  In  Year  1,  four  teams 
provided  both  the  ratings  from  the  independent  raters  and  the  consensus  ratings,  two  teams  provided  only  the 
independent  ratings,  and  one  team  provided  a single  rating.  In  Year  2,  all  teams  provided  the  requested  ratings. 

In  Year  3,  six  teams  provided  the  requested  ratings  and  one  team  provided  only  the  independent  ratings. 


42 


Chapter  1.  The  Social  and  Character  Development  Multiprogram  Evaluation 


was  sufficient  to  generate  precise  estimates.  When  applying  these  two  criteria  to  the  spring  2005  data, 
placement  into  the  high-  versus  low-fidelity  subgroup  was  defined  using  a cutoff  of  7 for  the  consensus  score: 
high  implementers  had  consensus  scores  that  were  7 or  greater;  and  low  implementers  had  consensus  scores 
less  than  7.  The  cutoff  value  of  7 led  to  the  highest  agreement  rate  between  the  raters  and  the  most  balanced 
division  of  the  sample. 16 

The  seven  rater  pairs  placed  30  of  the  36  schools  in  the  same  fidelity  group,  and  there  was  an  overall  kappa 
value  of  0.67. 17  Twenty  schools  were  categorized  as  high  implementers  and  22  were  categorized  as  low 
implementers.  Replication  of  this  procedure  with  the  2006  and  2007  data  yielded  similar  results,  including 
kappas  of  0.71  and  0.76,  respectively,  and  20  and  19  high-implementing  schools  and  22  and  23  low- 
implementing  schools,  respectively.  Table  1.7  shows  these  results  by  program  and  provides  the  inter -rater 
reliability  for  the  two  raters  in  each  program,  the  agreement  rate  between  raters  for  the  categorization  of 
schools  as  high  implementers,  and  Cohen’s  kappa  for  this  rate.  The  overall  inter -rater  reliability  coefficients 
(Cronbach’s  alpha)  were  0.71  in  Year  1 (0.83  if  the  LBW’s  alpha  of  -0.08  is  excluded),  0.90  in  Year  2,  and  0.94 
in  Year  3.  The  program-specific  alphas  were  0.70  or  higher  in  Year  1 (except  for  LBW  and  SS,  for  which  only 
a single  rating  was  provided),  0.74  or  higher  in  Year  2,  and  0.72  or  higher  in  Year  3.  These  findings  suggest 
that,  in  most  programs,  the  variance  of  ratings  across  observers  was  small  relative  to  the  variance  of  mean 
ratings  across  schools. 


16  Alternative  cutoff  values  led  to  lower  kappa  values  (almost  half  the  size)  and  less  balanced  groups. 

17  While  there  were  42  treatment  schools,  one  research  team  did  not  provide  rater-specific  information  for  the  spring 
2005  fidelity  data  so  there  were  only  36  schools  for  which  rater-specific  information  was  available. 
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Table  1.7.  Fidelity  ratings  of  program  implementation  at  treatment  schools 


Program 

Number  of 
treatment  schools 

Inter-rater  reliability 
(Cronbach’s  alpha) 

Rater  fidelity 
agreement  rate  (%) 

Cohen’s  kappa1 

Year  1 

Total 

42 

0.71 

83 

0.67 

ABC 

6 

1.00 

100 

1.00 

CSP 

5 

0.70 

80 

0.62 

LBW 

6 

-0.08 

67 

— 

PA 

7 

0.85 

86 

0.72 

PATHS 

5 

0.78 

80 

0.55 

4Rs 

7 

0.84 

86 

0.70 

SS2 

6 

— 

— 

— 

Year  2 

Total 

42 

0.90 

86 

0.71 

ABC 

6 

0.98 

100 

1.00 

CSP 

5 

0.80 

80 

0.55 

LBW 

6 

0.83 

100 

1.00 

PA 

7 

0.74 

71 

0.46 

PATHS 

5 

0.93 

60 

0.17 

4Rs 

7 

0.78 

86 

0.59 

SS 

6 

1.00 

100 

1.00 

Year  3 

Total 

42 

0.94 

88 

0.76 

ABC 

6 

0.99 

83 

0.67 

CSP 

5 

0.99 

100 

1.00 

LBW 

6 

0.97 

100 

1.00 

PA 

7 

0.98 

100 

1.00 

PATHS 

5 

0.86 

60 

0.29 

4Rs 

7 

0.72 

71 

0.30 

SS 

6 

0.99 

100 

1.00 

— Not  available. 

1 Cohen's  kappa  adjusts  the  agreement  rate  for  the  expected  number  of  Rater  1 and  Rater  2 exact  matches  that  would  occur 
because  of  chance. 

2 In  Year  1 , Second  Step  provided  a single  consensus  rating. 

NOTE:  Abbreviations  are 

ABC:  Academic  and  Behavioral  Competencies  Program 
CSP:  Competence  Support  Program 
LBW:  Love  In  a Big  World 
PA:  Positive  Action 

PATHS:  Promoting  Alternative  Thinking  Strategies 

4Rs:  The  4Rs  Program  (Reading,  Writing,  Respect,  and  Resolution) 

SS:  Second  Step 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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In  Year  1,  these  high-  or  low-fidelity  subgroups  were  used  direcdy  in  the  analysis.  In  Years  2 and  3,  the  actual 
subgroups  used  in  the  analysis  were  based  on  a school’s  assignment  to  the  high-  or  low-fidelity  subgroup  in 
each  year.  This  was  done  to  address  the  possibility  that  a program’s  fidelity  over  time  might  have  cumulative 
effects  on  students.  As  a result,  three  subgroups  were  created  for  Year  2 and  Year  3 by  combining  the  annual 
results:  (1)  high:  placement  in  the  high-fidelity  subgroup  in  all  years;  (2)  low:  placement  in  the  low-fidelity 
subgroup  in  all  years;  and  (3)  mixed:  placement  in  the  high-  and  low-fidelity  subgroups  in  different  years. 

The  multiprogram  evaluation  examined  seven  different  SACD  programs,  each  with  unique  features,  and  the 
fidelity  rating  measured  fidelity  of  implementation  relative  to  the  targets  established  for  each  specific 
intervention.18  In  this  way,  the  ratings  were  standardized  relative  to  each  site’s  program-specific  benchmarks 
and  could  be  compared  across  programs  and  years.  Thus,  for  example,  a rating  of  6 at  a school  using  one 
program  shows  higher  fidelity  to  implementation  of  that  program  than  a rating  of  4 at  another  school  using  a 
different  program  (relative  to  that  program’s  benchmarks).  The  compromise  in  developing  this  fidelity 
measure  was  that  basing  the  ratings  on  implementation  relative  to  each  program’s  standards  allows  for  cross- 
program comparisons  but  it  cannot  account  for  differences  among  the  programs’  implementation  standards. 
For  example,  one  SACD  program  might  be  simple  to  implement  and  so  require  a less  intensive 
implementation,  which  would  make  high  fidelity  easier  to  reach.  Another  program  might  be  very  difficult  to 
fully  implement  and  require  intensive  implementation,  which  would  make  high  fidelity  difficult.  As  a result, 
schools  using  the  first  program  might  score  higher  on  the  fidelity  rating  scale  than  schools  using  the  second 
program  because  high  fidelity  was  easier  for  them  to  obtain.  However,  if  the  second  program  was  more 
effective  than  the  first,  even  when  not  fully  implemented,  then  an  analysis  comparing  the  fidelity  subgroups 
might  erroneously  lead  to  the  conclusion  that  it  is  better  to  implement  programs  with  a low  level  of  fidelity. 

In  addition,  high  and  low  fidelity  were  not  randomly  assigned,  therefore  programs  and  schools  were  not 
necessarily  represented  in  the  fidelity  subgroups  by  chance.  Instead,  they  may  have  ended  up  in  their 
subgroup  because  of  pre-existing  characteristics.  Comparisons  of  the  estimated  impacts  at  high-fidelity 
schools  to  those  of  low-fidelity  schools  would  lead  to  an  erroneous  conclusion  regarding  the  effect  of  fidelity 
of  implementation  if  the  impact  of  fidelity  was  due  to  other  differences  between  the  subgroups.  In  other 
words,  if  high  and  low  fidelity  occurred  because  of  differences  among  schools  and  these  differences  also  had 
an  impact  on  the  student  outcomes,  then  fidelity  would  be  an  outcome  of  these  differences  between  the 
schools  rather  than  a cause  of  student  outcomes.  These  differences  might  be  more  observable  (e.g.,  schools 
with  high  fidelity  had  students  who,  on  average,  had  a different  socioeconomic  status  than  schools  with  low 
fidelity)  or  less  observable  (e.g.,  differences  in  the  talent  and  dedication  of  the  staff  or  strength  of  school 
leadership). 

In  sum,  the  fidelity  ratings  provided  an  indicator  of  the  extent  to  which  the  SACD  programs  were 
implemented  according  to  program  benchmarks.  Because  of  how  the  ratings  were  constructed  and  the 
possibility  that  program  and  school  characteristics  may  have  contributed  to  each  school’s  rating,  differences 
in  estimated  impacts  by  implementation  group  could  reflect  differences  in  the  level  of  implementation  and/ or 
underlying  differences  between  the  students,  the  schools,  and  the  SACD  programs.  As  these  effects  cannot 
be  fully  disentangled,  the  analysis  making  use  of  the  fidelity  data  was  considered  exploratory  rather  than 
definitive. 

Data  Collection 

The  SACD  study  had  five  data  collection  points,  beginning  with  the  initial  collection  of  data  in  fall  2004  (start 
of  third  grade)  and  followed  by  four  follow-up  points:  spring  2005  (end  of  third  grade),  fall  2005  (start  of 


18  One  drawback  of  this  approach  is  that  the  fidelity  data  provide  little  information  about  why  implementation  was  of  a 
certain  quality.  The  fidelity  data  collected  by  each  research  team  using  team-chosen  criteria  were  expected  to  provide  this 
type  of  information  rather  than  facilitate  the  multiprogram  analysis. 
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fourth  grade),  spring  2006  (end  of  fourth  grade)  and  spring  2007  (end  of  fifth  grade).19  Ideally,  fall  data 
collections  would  have  occurred  at  the  very  beginning  of  the  school  year  to  reduce  the  possibility  that  the 
implementation  of  the  program  could  affect  the  data  (for  fall  2004)  and  to  maximize  the  length  of  time  that 
the  intervention  would  have  to  affect  the  annually  collected  outcomes.  Spring  data  collections  would  ideally 
have  occurred  near  the  end  of  the  school  year  in  order  to  maximize  the  length  of  time  the  program  had  to 
affect  annually  collected  outcomes. 

For  practical  reasons,  fall  data  collections  were  often  delayed  for  several  weeks  to  allow  school  populations  to 
settle,  to  obtain  primary  caregiver  consent,  and  to  avoid  disrupting  planned  school  activities.  Spring  data 
collections  occurred  earlier  than  the  end  of  school  to  fit  into  school  schedules  that  included  spring  district 
testing  and  vacations. 

Across  the  seven  SACD  programs,  the  fall  2004  data  collection  encompassed  a 3 -month  period,  although  on- 
site data  collection  in  each  program  typically  took  2 to  4 weeks  (table  1.8).  The  earliest  fall  2004  data 
collection  started  on  August  23,  2004  (LBW),  and  the  last  ended  on  November  12,  2004  (ABC).  In  all 
programs  except  LBW,  implementation  of  the  program  started  before  the  fall  2004  data  collection  visit.  In 
the  other  six  programs,  the  interval  between  program  implementation  and  fall  2004  data  collection  was  2 to  6 
weeks,  which  may  have  had  some  effect  on  student  reports.  Teachers  and  principals  received  training  on  the 
intervention  before  the  fall  2004  data  collection,  and  this  could  have  affected  their  reports  on  the  use  of 
activities  to  promote  social  and  character  development  in  their  classrooms  and  schools.  As  a result,  the  fall 
2004  reports,  especially  those  from  teachers  and  principals,  are  unlikely  to  reflect  the  true  pre -intervention 
conditions  but  instead  capture  what  was  being  done  near  the  beginning  of  the  evaluation.  For  this  reason,  the 
fall  2004  data  are  termed  “initial  data”  rather  than  “baseline  data.” 

The  spring  2005  data  follow-up  occurred  between  March  21  and  June  3,  with  most  of  it  concentrated  in  April 
and  May.  On-site  data  collection  in  each  program  typically  took  2 to  4 weeks.  Because  of  differences  in  the 
start  dates  of  implementation  and  the  dates  of  spring  data  collection,  there  was  variation  among  the  programs 
in  terms  of  how  long  they  had  been  implemented  by  the  time  of  the  spring  data  collection.  This  interval 
ranged  from  a low  of  26  weeks  for  4Rs  to  a high  of  36  weeks  for  CSP.  These  differences  should  be  kept  in 
mind  when  viewing  the  results  of  the  analysis  comparing  the  outcomes  of  students  at  spring  2005  in 
treatment  schools  with  those  of  students  at  spring  2005  in  control  schools  (while  controlling  for  the  fall  value 
of  the  relevant  outcome  measure),  although  they  did  not  affect  individual  program  evaluations  as  each 
program’s  treatment  and  control  group  was  compared  over  the  same  period. 

The  fall  2005  data  collection  encompassed  a 3-month  period  from  August  to  November,  with  on-site  data 
collection  in  each  program  typically  taking  3 to  4 weeks.  The  gap  between  the  start  of  school  and  the  fall  data 
collection  was  5 weeks  for  six  of  the  programs  and  8 weeks  for  the  seventh,  SS.  The  2006  spring  data 
collection  lasted  for  approximately  a 2-month  period,  occurring  between  March  20,  2006,  and  May  26,  2006. 
The  length  of  the  data  collection  ranged  from  less  than  2 weeks  to  about  6 weeks.  The  time  between  the  start 
of  the  school  year  and  the  spring  data  collection  varied  from  29  to  40  weeks  across  the  programs. 

The  spring  2007  data  collection  lasted  approximately  2 months  and  occurred  between  March  26  and  June  1. 
On-site  data  collection  in  each  program  took  2 to  5 weeks.  The  time  between  the  start  of  the  school  year  and 
spring  data  collection  ranged  from  29  to  40  weeks,  similar  to  Year  2. 


19  Data  collection  for  the  second  cohort  (discussed  in  appendix  A)  began  in  fall  2005  (start  of  third  grade)  with  follow- 
ups in  spring  2006  (end  of  third  grade),  and  spring  2007  (end  of  fourth  grade). 
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Table  1.8.  Period  between  implementation  and  data  collection  and  between  data  collections,  by  program 


Data  collection  schedule 

ABC 

CSP 

LBW 

PA 

PATHS 

4Rs 

SS 

Fall  2004  data  collection 

Start  date 

10/18/041 

8/30/04 

8/23/04 

10/12/04 

10/18/04 

10/18/04 

10/11/04 

End  date 

11/12/04 

9/16/04 

9/9/04 

10/28/04 

1 0/28/04 

11/10/04 

1 0/29/04 

Calendar  weeks  from  program  implementation 

6 

4 

t2 

2 

6 

4 

5 

Spring  2005  data  collection 

Start  date 

4/11/05 

4/11/05 

5/2/05 

5/1 6/05 

3/28/05 

3/21/05 

4/25/05 

End  date 

5/6/05 

4/29/05 

5/20/05 

6/3/05 

4/28/05 

5/5/05 

5/1 3/05 

Calendar  weeks  from  end  of  fall  data  collection 

21 

30 

34 

29 

22 

17 

25 

Calendar  weeks  from  program  implementation 

31 

36 

33 

33 

29 

26 

33 

Fall  2005  data  collection 

Start  date 

10/10/051 

9/26/053 

8/29/05 

10/11/05 

10/11/05 

10/11/05 

1 0/24/05 

End  date 

10/27/05 

10/20/05 

9/23/05 

11/2/05 

1 0/27/05 

11/16/05 

11/11/05 

Calendar  weeks  from  start  of  school 

5 

5 

54 

5 

5 

5 

8 

Spring  2006  data  collection 

Start  date 

3/27/06 

4/24/06 

5/1/06 

5/1/06 

3/20/06 

3/27/06 

4/24/06 

End  date 

5/1/06 

5/5/06 

5/19/06 

5/26/06 

4/11/06 

5/8/06 

5/1 2/06 

Calendar  weeks  from  end  of  fall  data  collection 

21 

27 

31 

26 

20 

20 

23 

Calendar  weeks  from  start  of  school 

29 

35 

40 

34 

29 

29 

34 

See  notes  at  end  of  table. 
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Table  1.8.  Period  between  implementation  and  data  collection  and  between  data  collections,  by  program — Continued 


Data  collection  schedule 

ABC 

CSP 

LBW 

PA 

PATHS 

4Rs 

SS 

Spring  2007  data  collection 

Start  date 

4/1 8/07 

4/26/07 

5/1/07 

5/7/07 

3/26/07 

3/26/07 

4/23/07 

End  date 

5/8/07 

5/10/07 

5/1 6/07 

6/1/07 

4/27/07 

4/26/07 

5/1 0/07 

Calendar  weeks  from  start  of  school 

325 

35 

40s 

35 

29 

29 

34 

t Not  applicable. 

1 At  two  schools  data  collection  began  the  week  of  9/20  in  2004  and  2005. 

2 Data  collection  occurred  before  program  implementation. 

3 Data  collection  at  one  school  occurred  3 weeks  earlier. 

4 One-third  of  the  schools  started  2 weeks  later. 

5 Two  schools  started  several  weeks  earlier. 

6 Two  schools  started  2 weeks  later. 

NOTE:  Abbreviations  are 

ABC:  Academic  and  Behavioral  Competencies  Program 
CSP:  Competence  Support  Program 
LBW:  Love  In  a Big  World 
PA:  Positive  Action 

PATHS:  Promoting  Alternative  Thinking  Strategies 
4Rs:  The  4Rs  Program  (Reading,  Writing,  Respect,  and  Resolution) 
o°  SS : Second  Step 

There  may  be  slight  discrepancies  in  the  end  dates  for  field  data  collection  due  to  scheduling  of  Child  Report  make-up  sessions  and  principal  interviews. 
SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Training  of  Data  Collectors 

Data  collection  was  done  by  MPR  (along  with  its  subcontractor  Decision  Information  Resources,  Inc.  [DIR]) 
and  the  research  teams.  MPR  established  on-site  data  collection  teams  to  administer  and  collect  the  Child 
Report  from  the  study  schools.  These  teams  were  trained  by  senior  MPR  and  DIR  data  collection  managers 
at  one  of  two  4-day  training  sessions  held  August  7-10,  2004,  in  Tennessee  and  September  18-21,  2004,  in 
New  Jersey.  At  these  sessions,  data  collectors  were  introduced  to  the  study  objectives,  reviewed  data 
collection  procedures,  discussed  working  in  schools,  and  reviewed  the  distribution  and  collection  of  all  survey 
materials.  To  monitor  quality  and  consistency  of  data  collection  across  all  sites,  senior  MPR  and  DIR  data 
collection  managers  observed  the  first  day  or  two  of  on-site  data  collection  at  each  site  in  fall  2004  and  spring 
2005.  Because  most  of  the  data  collectors  from  fall  2004  returned  for  spring  2005  data  collection,  MPR  held  a 
refresher  training  session  in  spring  2005.  This  refresher  training  included  several  activities:  (1)  distribution  of  a 
revised  training  manual,  (2)  a videotape  and  detailed  memorandum  on  changes  in  procedures  and/ or 
instruments  since  fall  data  collection  for  home  study,  (3)  a telephone  conference  call  to  review  home  study 
materials,  and  (4)  an  in-person  half-day  meeting  a few  days  before  the  beginning  of  data  collection.  Ninety- 
three  percent  of  the  collectors  returned  for  the  fall  2005  data  collection;  at  one  site  it  proved  difficult  to  hire 
new  collectors  so  a team  was  composed  of  collectors  from  other  sites.  MPR  provided  training  through  a 
review  memo  mailed  to  collectors,  conference  calls,  and  a 4-hour  refresher  course  with  each  team  prior  to 
data  collection.  Again  an  MPR  or  DIR  representative  worked  with  each  team  for  the  first  day  or  two  of  data 
collection.  By  spring  2006,  turnover  was  high  enough  that  new  collectors  were  provided  a 2-day  training  in 
March  or  April.  Each  team  also  went  through  the  4-hour  refresher  course  prior  to  data  collection.  A similar 
approach  was  taken  for  spring  2007,  with  a 2-day  training  for  new  data  collectors  and  a 4-hour  refresher 
course  before  data  collection. 

The  two  teacher  reports  and  the  primary  caregiver  report  were  self-administered  questionnaires  prepared  by 
MPR.  They  were  distributed  either  by  MPR  or  the  research  team  during  on-site  data  collection  (in  some  sites 
teachers  received  their  questionnaires  before  spring  on-site  data  collection).  Teachers  received  their  surveys 
directly  and  sent  the  primary  caregiver  surveys  home  with  students.  If  a completed  Primary  Caregiver  Report 
was  not  returned  within  2 to  4 weeks,  MPR  attempted  to  contact  the  child’s  primary  caregiver  by  telephone, 
and  if  the  primary  caregiver  was  willing,  the  Primary  Caregiver  Report  was  administered  as  a computer- 
assisted  telephone  interview. 

Consent  Rates,  Completion  Rates,  and  Percentage  of  Sample  With  Data 

Data  collection  required  a two-part  process:  consent  and  completion  of  data  reports.  First,  written  consent 
had  to  be  obtained  from  primary  caregivers  before  their  children  could  fill  out  the  Child  Report,  their 
children’s  teachers  could  fill  out  the  Teacher  Report  on  Student,  and  they  themselves  could  fill  out  the 
Primary  Caregiver  Report.  Written  consent  from  teachers  had  to  be  obtained  before  they  could  fill  out  the 
Teacher  Report  on  Classroom  and  School.  Second,  even  when  consent  was  received,  reports  might  not  be 
completed  for  other  reasons.  Students  may  have  decided  not  to  complete  the  Child  Report  or  were  absent  on 
the  days  of  the  data  collection  and  make-up  sessions.  Primary  caregivers  may  have  failed  to  complete  the 
Primary  Caregiver  Report  or  did  not  complete  the  follow-up  phone  interview.  Teachers  may  have  failed  to 
complete  the  Teacher  Report  on  Student  or  the  Teacher  Report  on  Classroom  and  School. 

The  percentage  of  the  sample  for  which  there  are  data  is  calculated  by  multiplying  the  consent  rate  and  the 
completion  rate  for  each  report.  This  section  discusses  the  percentage  of  the  sample  for  which  there  are  data 
as  follows:  (1)  the  overall  sample,  which  includes  treatment  and  control  groups;  (2)  students  from  the  original 
cohort  (stayers)  versus  new  entrants;  and  (3)  by  individual  SACD  program.  The  consent  and  completion  rates 
are  also  provided  for  the  first  two  and  are  available  in  chapters  2 to  8 by  program. 

Table  1.9  presents  the  percentage  of  sample  for  which  there  are  data  for  each  report  by  year  and  by  treatment 
versus  control  group.  For  the  Child  Report  and  the  Teacher  Report  on  Student,  between  61  percent  and  65 
percent  of  students  have  data  over  the  3 years.  In  Year  1 (third  grade),  the  treatment  group  had  statistically 
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significantly  larger  percentages  of  data  but  this  was  not  the  case  in  Years  2 and  3.  For  the  Primary  Caregiver 
Report,  the  percentage  of  primary  caregivers  with  data  ranged  from  57  percent  in  Year  1 to  46  percent  in 
Year  3,  due  to  declines  in  completion  rates.  The  treatment  group  had  a significantly  higher  percentage  of 
primary  caregivers  with  data  in  the  fall  of  Year  1 but  not  thereafter.  For  the  Teacher  Report  on  Classroom 
and  Schools,  the  percentage  of  teachers  with  data  ranged  from  87  percent  to  90  percent  over  the  3 years,  with 
no  significant  difference  between  the  treatment  and  control  schools. 

Table  1.10  examines  differences  for  stayers  versus  new  entrants.  For  all  three  reports  on  students  (student, 
primary  caregiver,  and  teacher)  stayers  had  statistically  significantly  higher  percentages  of  data  than  new 
entrants  due  to  higher  consent  rates. 

Table  1.11  presents  the  percentage  of  sample  with  data  for  treatment  versus  control  group  by  SACD 
program.  Out  of  112  comparisons,  18  showed  statistically  significant  differences  (6  such  differences  would  be 
expected  by  chance)  with  all  but  2 favoring  the  treatment  group.  These  break  out  as  6 differences  in  the  Child 
Report,  3 differences  in  the  Primary  Caregiver  Report,  7 differences  in  the  Teacher  Report  on  Student,  and  2 
differences  in  the  Teacher  Report  on  Classroom  and  School. 

In  sum,  over  the  3 years  of  the  SACD  evaluation,  61  percent  to  65  percent  of  the  student  sample  had  data 
(with  46%  to  57%  having  primary  caregiver  data).  20 


20  The  majority  of  the  students  without  data  (67%  to  97%,  depending  on  report)  did  not  have  consent  to  take  part  in  the 
study.  As  data  could  not  be  collected  on  these  students,  no  comparisons  could  be  made  to  determine  whether  they 
significantly  differed  from  those  taking  part  in  the  study. 
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Table  1.9.  Consent  rates,  completion  rates,  and  percentage  of  sample  with  data  from  each  report 


Year  1 Year  1 Year  2 Year  3 

(Fall  3rd  grade)  (Spring  3rd  grade)  (Spring  4th  grade)  (Spring  5th  grade) 

Treat-  Treat-  Treat-  Treat- 


Report 

Total 

ment  Control 

Total 

ment 

Control 

Total 

ment 

Control 

Total 

ment 

Control 

Student  sample  size 

6,567 

3,367 

3,200 

6,597 

3,388 

3,209 

6,415 

3,327 

3,088 

6,249 

3,172 

3,077 

Child  Report  (percent) 

Primary  caregiver  consent  rate 

65 

67** 

63 

66 

68** 

64 

67 

67 

66 

66 

67 

66 

Student  completion  rate 

94 

93* 

94 

96 

96 

96 

95 

96 

95 

96 

97 

96 

Students  with  data1 

61 

62* 

60 

63 

65** 

61 

63 

65 

62 

64 

65 

63 

Primary  Caregiver  Report  (percent) 

Primary  caregiver  consent  rate 

63 

64** 

61 

64 

66** 

62 

64 

65 

63 

64 

65 

64 

Primary  caregiver  completion  rate 

92 

92 

92 

80 

80 

81 

78 

78 

77 

72 

71 

72 

Primary  caregiver  with  data1 

57 

59* 

56 

51 

52 

50 

50 

51 

49 

46 

46 

46 

Teacher  Report  on  Student  (percent) 

Primary  caregiver  consent  rate2 

65 

67** 

63 

66 

68** 

64 

67 

67 

66 

66 

67 

66 

Teacher  completion  rate 

96 

96 

96 

99 

99 

99 

100 

100** 

99 

98 

98 

99 

Students  with  data1 

62 

64** 

61 

65 

67** 

63 

66 

67 

65 

65 

66 

65 

Teacher  Report  on  Classroom  and  School 
(3rd-  to  5th-grade  teachers)  (percent) 

Teacher  consent  rate 

96 

98*** 

92 

95 

97* 

94 

95 

97 

94 

96 

97 

95 

Teacher  completion  rate 

91 

90 

93 

91 

90 

91 

94 

94 

94 

92 

91 

93 

Teachers  with  data1 

87 

88 

86 

87 

88 

86 

90 

90 

89 

89 

88 

89 

* Treatment  group  significantly  different  from  control  group  at  the  .05  level. 

**  Treatment  group  significantly  different  from  control  group  at  the  .01  level. 

***  Treatment  group  significantly  different  from  control  group  at  the  .001  level. 

1 Calculated  as  consent  rate  x completion  rate. 

2 The  primary  caregiver  consent  rates  for  the  Child  Report  and  the  Teacher  Report  on  Student  are  identical,  as  the  primary  caregiver  gave  consent  to  both  together. 
SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Table  1.10.  Consent  rates,  completion  rates,  and  percentage  of  sample  with  data:  Stayers  versus  new  entrants 


Year  1 Year  2 Year  3 

(Spring  3rd  grade)  (Spring  4th  grade)  (Spring  5th  grade) 

New  New  New 


Report 

Total 

Stayers 

entrants 

Total 

Stayers 

entrants 

Total 

Stayers 

entrants 

Student  sample  size 

6,597 

6,203 

394 

6,415 

5,110 

1,305 

6,249 

4,500 

1,749 

Child  Report  (percent) 

Primary  caregiver  consent  rate 

65.9 

66.5*** 

56.1 

66.5 

69.4*** 

55.2 

66.4 

69.1*** 

59.7 

Student  completion  rate 

95.9 

96.0 

94.6 

95.4 

95.2 

96.7 

96.2 

96.1 

96.4 

Students  with  data1 

63.2 

63.8*** 

53.0 

63.5 

66.0*** 

53.4 

63.9 

66.4*** 

57.5 

Primary  Caregiver  Report  (percent) 

Primary  caregiver  consent  rate 

63.8 

64.5*** 

53.8 

64.2 

66.7*** 

54.1 

64.2 

66.3*** 

58.8 

Primary  caregiver  completion  rate 

80.3 

80.2 

81.6 

77.7 

77.5 

78.6 

71.7 

70.7* 

74.5 

Primary  caregivers  with  data1 

51.3 

51.7** 

43.9 

49.9 

51.7*** 

42.5 

46.0 

46.9* 

43.8 

Teacher  Report  on  Student  (percent) 

Primary  caregiver  consent  rate2 

65.9 

66.5*** 

56.1 

66.5 

69.4*** 

55.2 

66.4 

69.1*** 

59.7 

Teacher  completion  rate 

98.8 

98.9* 

97.3 

99.7 

99.7 

99.4 

98.5 

98.6 

98.0 

Students  with  data1 

65.1 

65.7*** 

54.6 

66.3 

69.2*** 

54.9 

65.4 

68.1*** 

58.5 

* Treatment  group  significantly  different  from  control  group  at  the  .05  level. 

**  Treatment  group  significantly  different  from  control  group  at  the  .01  level. 

***  Treatment  group  significantly  different  from  control  group  at  the  .001  level. 

1 Calculated  as  consent  rate  x completion  rate. 

2 The  primary  caregiver  consent  rates  for  the  Child  Report  and  the  Teacher  Report  on  Student  are  identical,  as  the  primary  caregiver  gave  consent  to  both  together. 
SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Table  1.11.  Percentage  of  sample  with  data  by  report  and  program1 

Year  1 Year  1 Year  2 Year  3 

(Fall  3rd  grade)  (Spring  3rd  grade)  (Spring  4th  grade)  (Spring  5th  grade) 


Treat-  Treat-  Treat-  Treat- 


Report 

Total 

ment 

Control 

Total 

ment 

Control 

Total 

ment 

Control 

Total 

ment 

Control 

Child  Report 

ABC 

59.4 

62.7* 

56.7 

62.5 

66.0 

60.0 

62.3 

62.7 

62.0 

63.4 

69.1** 

59.5 

CSP 

64.8 

62.8 

66.6 

67.8 

67.2 

68.4 

66.5 

66.2 

66.7 

65.4 

59.2*** 

71.2 

LBW 

60.1 

61.5 

58.4 

60.8 

61.4 

60.0 

60.4 

60.4 

60.3 

58.6 

59.1 

57.8 

PA 

74.4 

73.9 

74.8 

71.2 

71.6 

70.7 

66.6 

66.8 

66.4 

76.5 

78.3 

74.7 

PATHS 

52.2 

54.4 

50.1 

56.1 

58.6 

53.8 

55.8 

59.0 

52.8 

55.4 

59.5* 

51.5 

4Rs 

54.9 

56.0 

53.6 

59.1 

62.3* 

55.4 

61.2 

62.8 

59.4 

62.4 

63.5 

61.3 

SS 

62.3 

65.7* 

58.0 

65.6 

68.4 

62.2 

71.0 

72.4 

69.2 

68.0 

70.4 

65.4 

Primary  Caregiver  Report 

ABC 

56.6 

60.0 

53.9 

48.1 

52.5 

44.8 

45.8 

45.8 

45.9 

44.3 

47.3 

42.3 

CSP 

59.6 

56.3* 

62.9 

53.9 

50.9 

56.9 

52.1 

50.2 

53.9 

46.7 

43.4 

49.7 

LBW 

62.6 

64.8 

59.8 

53.1 

54.0 

52.0 

51.2 

52.9 

48.9 

46.9 

45.7 

48.8 

PA 

72.3 

72.7 

71.8 

58.9 

58.9 

58.8 

52.2 

50.6 

54.3 

50.5 

51.7 

49.4 

PATHS 

53.6 

56.0 

51.3 

48.3 

52.9 

44.0 

45.5 

49.6** 

41.7 

43.4 

45.2 

41.8 

4Rs 

41.0 

42.2 

39.6 

41.7 

41.9 

41.5 

41.0 

40.9 

41.2 

33.1 

32.0 

34.4 

SS 

62.3 

65.7* 

57.8 

57.3 

58.0 

56.5 

61.8 

63.2 

60.1 

56.7 

58.3 

54.8 

Teacher  Report  on  Student 

ABC 

59.7 

65.3** 

55.5 

63.7 

66.8 

61.4 

64.7 

65.4 

64.1 

67.2 

73.7* 

62.7 

CSP 

66.4 

64.1 

68.7 

68.7 

67.8 

69.6 

68.3 

67.3 

69.3 

62.8 

55.0*** 

70.1 

LBW 

64.3 

66.3 

61.9 

66.1 

67.8 

63.8 

64.4 

65.5 

63.0 

63.7 

63.8 

63.4 

PA 

74.6 

72.9 

76.3 

74.8 

73.3 

76.3 

72.4 

71.1 

74.0 

78.3 

79.5 

77.1 

PATHS 

56.4 

58.6 

54.3 

58.4 

62.0* 

55.0 

57.6 

60.9 

54.6 

56.3 

60.3* 

52.5 

4Rs 

53.8 

55.5 

51.8 

59.6 

62.4* 

56.3 

65.5 

67.3 

63.3 

64.1 

64.4 

63.9 

SS 

64.8 

68.5** 

60.2 

65.6 

68.4 

62.2 

70.8 

72.0 

69.2 

68.1 

70.4 

65.6 

See  notes  at  end  of  table. 
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Table  1.11.  Percentage  of  sample  with  data  by  report  and  program1 — Continued 


Year  1 Year  1 Year  2 Year  3 

(Fall  3rd  grade)  (Spring  3rd  grade)  (Spring  4th  grade)  (Spring  5th  grade) 


Treat-  Treat-  Treat-  Treat- 


Report 

Total 

ment 

Control 

Total 

ment 

Control 

Total 

ment 

Control 

Total 

ment 

Control 

Teacher  Report  on  Classroom  and 
School  (3rd-  to  5th-grade  teachers) 
ABC 

85.6 

91.8 

80.8 

84.2 

83.6 

84.6 

86.0 

84.7 

87.0 

81.6 

86.7 

77.8 

CSP 

84.1 

83.1 

85.2 

78.5 

77.9 

79.0 

88.0 

85.1 

90.9 

90.6 

86.3 

95.4 

LBW 

88.4 

84.0 

93.8 

95.9 

95.2 

96.9 

87.9 

89.7 

85.2 

91.9 

90.4 

94.3 

PA 

89.5 

90.0 

88.9 

82.6 

82.0 

83.3 

86.3 

86.0 

86.7 

83.7 

78.0 

90.5 

PATHS 

78.7 

88.9** 

69.9 

80.1 

92.1** 

69.9 

83.6 

90.0 

78.4 

83.5 

81.7 

84.9 

4Rs 

92.7 

92.0 

93.4 

90.6 

89.0 

92.4 

97.2 

96.8 

97.6 

96.4 

97.8 

94.9 

SS 

92.2 

90.3 

94.4 

93.1 

93.5 

92.6 

96.6 

96.8 

96.2 

89.0 

90.8 

86.8 

* Treatment  group  significantly  different  from  control  group  at  the  .05  level. 

**  Treatment  group  significantly  different  from  control  group  at  the  .01  level. 

***  Treatment  group  significantly  different  from  control  group  at  the  .001  level. 

1 Calculated  as  consent  rate  x completion  rate. 
c_n  NOTE:  Abbreviations  are 

ABC:  Academic  and  Behavioral  Competencies  Program 
CSP:  Competence  Support  Program 
LBW:  Love  In  a Big  World 
PA:  Positive  Action 

PATHS:  Promoting  Alternative  Thinking  Strategies 

4Rs:  The  4Rs  Program  (Reading,  Writing,  Respect,  and  Resolution) 

SS:  Second  Step 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Initial  Characteristics 

This  section  examines  the  initial  characteristics  of  the  students,  primary  caregivers,  teachers,  and  schools 
participating  in  the  SACD  evaluation.  These  characteristics  were  collected  from  students  who  were  enrolled 
in  the  third  grade  at  the  study  schools  in  fall  2004,  as  well  as  from  their  primary  caregivers  and  third-grade 
teachers.  In  addition,  third-,  fourth-,  and  fifth-grade  teachers  and  principals  in  the  study  schools  provided 
information  about  activities  related  to  social  and  character  development  in  these  schools.  Documenting  the 
characteristics  of  students,  primary  caregivers  and  the  students’  households,  teachers,  and  schools — and  initial 
measures  of  key  outcomes  at  a point  before  the  programs  had  been  operating  for  an  extended  period — helps 
to  determine  whether  the  random  assignment  of  schools  to  treatment  and  control  status  produced  treatment 
and  control  groups  with  similar  distributions  of  observed  characteristics. 

For  the  treatment  schools  that  were  implementing  six  of  the  seven  programs  (36  of  the  42  treatment  schools), 
implementation  began  2 to  6 weeks  before  the  fall  2004  data  were  collected  (as  shown  in  table  1.8).  The 
implementation  of  the  six  programs,  and  the  training  of  teachers  that  preceded  the  implementation  of  all 
seven  programs,  could  have  affected  the  student,  primary  caregiver,  and  teacher  reports  from  the  treatment 
schools.  If  such  effects  occurred,  they  could  not  be  disentangled  from  pre-existing  differences  between  the 
treatment  and  control  schools.  The  likelihood  of  an  implementation  effect  differed  by  the  type  of 
characteristic  reported  on.  Some  characteristics  could  not  be  affected  (e.g.,  gender  and  race/ethnicity).  Some 
were  unlikely  to  be  affected  (e.g.,  primary  caregiver’s  highest  education  level)  and  others  could  have  been 
affected  (e.g.,  student  outcomes  and  risk  factors)  but  the  likelihood  was  low  given  how  short  the  time  the 
students  were  exposed  to  the  programs  before  the  data  collection.  Other  characteristics  were  likely  to  have 
been  affected  (e.g.,  teacher  reports  of  SACD  activities  and  professional  development  over  the  past  year)  given 
that  teachers  prepared  for  implementation  and  implemented  the  programs  before  data  collection.  How  likely 
a characteristic  was  to  have  been  affected  should  be  kept  in  mind  when  considering  how  similar  the  treatment 
and  control  groups  appeared  from  the  fall  2004  data  collection.  For  example,  the  lack  of  significant 
differences  in  family  characteristics  discussed  below  appears  to  provide  valid  evidence  that  the  treatment  and 
control  groups  were  similar,  while  the  lack  of  differences  in  student  outcomes  and  risk  factors  gives  similar 
but  perhaps  less  valid  evidence.  On  the  other  hand,  the  findings  that  treatment  teachers  used  more  SACD 
activities  in  the  classroom  and  received  more  SACD  training  provide  evidence  of  a difference  between  the 
treatment  and  control  schools  before  the  study  began,  but  the  training  in  and  implementation  of  the 
programs  before  data  collection  lead  to  less  confidence  in  this  finding. 

Characteristics  of  Children,  Their  Families,  and  Communities 

The  sample’s  treatment  and  control  groups  were  similar  along  the  observed  student,  primary  caregiver,  and 
community  characteristics  (table  1.12).  For  the  comparison  of  21  characteristics,  the  expectation  would  be 
that  1 would  be  significant  by  chance.  For  all  seven  programs  together,  there  was  only  1 statistically  significant 
difference  (treatment  group  students  were  more  likely  to  come  from  smaller  households  than  control  group 
students:  4.5  versus  4.7  people  per  household).  At  the  program  level,  there  was  only  1 statistically  significant 
difference  (control  group  primary  caregivers  had  significantly  higher  levels  of  education  than  treatment  group 
primary  caregivers  for  the  SS  program)  versus  the  7 that  would  be  expected  by  chance.21  For  students,  the 
mean  age  was  8 years.  The  sample  contained  more  girls  than  boys  (53%  versus  48%,  respectively)  but  the 
difference  was  not  statistically  significant.  This  pattern  prevailed  in  five  of  the  seven  programs.  The  sample 
was  ethnically  diverse.  White  non-Hispanic  students  made  up  42  percent  of  the  sample,  Black  non-Hispanic 
students  made  up  31  percent,  and  Hispanic  students  made  up  19  percent.  In  four  programs,  a single  ethnic 
group  comprised  the  majority  of  all  students  (White  non-Hispanics  made  up  83%  of  SS  students,  65%  of 
LBW  students,  and  56%  of  PATHS  students,  while  51%  of  PA  students  were  Black  non-Hispanic). 


21  Program-level  results  are  detailed  in  the  program-specific  chapters. 
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The  sample  was  also  diverse  in  levels  of  family  income,  education  levels  of  primary  caregivers,  and  family 
situation.  Thirty-nine  percent  of  children  lived  in  a household  in  which  the  income  was  135  percent  of  the 
federal  poverty  level  or  lower,  which  is  the  income  threshold  for  eligibility  for  free  school  meals.  About  15 
percent  of  primary  caregivers  had  not  completed  high  school.  Sixty  percent  of  the  children  lived  with  both 
their  father  and  their  mother.  Programs  varied  in  their  economic  diversity.  Three  of  the  programs  operated  in 
schools  in  which  the  majority  of  children  were  in  families  that  had  incomes  below  135  percent  of  the  federal 
poverty  level  (66%  for  PA,  58%  for  4Rs,  and  50%  for  ABC).  These  three  programs  also  had  between  47 
percent  and  50  percent  of  children  living  with  both  parents;  27  percent  to  28  percent  of  primary  caregivers 
lacked  a high  school  diploma  in  4Rs  and  PA.  On  the  other  hand,  for  SS,  only  5 percent  of  children  were  from 
families  with  incomes  below  135  percent  of  the  federal  poverty  level,  only  4 percent  of  primary  caregivers 
reported  not  having  a high  school  diploma,  and  80  percent  of  children  lived  with  both  parents. 
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Table  1.12.  Initial  characteristics  of  children,  their  families,  and  communities 


Characteristic 

Total 

Treatment 

Control 

Student  sample  size 

3,774 

1,980 

1,794 

Student  demographics 

Gender  (percent) 

Male 

47.5 

46.9 

48.1 

Female 

52.5 

53.1 

51.9 

Race/ethnicity  (percent) 

White  (non-Hispanic) 

42.1 

39.7 

44.5 

Black  (non-Hispanic) 

31.0 

34.2 

27.9 

Hispanic 

19.2 

18.5 

20.0 

Other 

7.7 

7.7 

7.6 

Age  (in  years)  (mean) 

8.1 

8.1 

8.1 

Primary  caregiver  and  family  characteristics 

Primary  caregiver’s  age  (in  years)  (mean) 

36.0 

36.0 

36.0 

Primary  caregiver’s  race/ethnicity  (percent) 

White  (non-Hispanic) 

46.1 

43.8 

48.4 

Black  (non-Hispanic) 

30.7 

33.8 

27.6 

Hispanic 

17.6 

16.9 

18.3 

Other 

5.6 

5.5 

5.6 

Primary  caregiver’s  education  (percent) 

Did  not  complete  high  school 

14.7 

13.5 

15.9 

Completed  high  school  or  equivalent 

24.3 

23.3 

25.4 

Some  college 

39.7 

41.3 

38.1 

Bachelor’s  or  higher  degree 

21.2 

21.9 

20.5 

Primary  caregiver’s  employment  (percent) 

Full-time 

48.3 

48.6 

48.0 

Part-time 

14.7 

14.5 

14.9 

Student  and  employed 

4.6 

5.1 

4.1 

Not  employed 

31.3 

30.8 

31.8 

Other 

1.1 

1.0 

1.2 

Primary  caregiver’s  marital  status  (percent) 

Single 

23.9 

24.5 

23.2 

Married 

57.0 

56.4 

57.7 

Separated  or  divorced 

12.7 

12.6 

12.9 

Widowed 

1.2 

1.2 

1.1 

Other1 

5.2 

5.2 

5.2 

Students  who  live  in  one  household  (percent) 

93.6 

93.6 

93.6 

Number  of  individuals  in  household  (mean) 

4.6 

4.5* 

4.7 

See  notes  at  end  of  table. 
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Table  1.12.  Initial  characteristics  of  children,  their  families,  and  communities — Continued 


Characteristic 

Total 

Treatment 

Control 

Primary  caregiver’s  relationship  to  child  (percent) 

Mother  (stepmother) 

86.1 

86.3 

85.9 

Father  (stepfather) 

9.0 

8.1 

9.8 

Grandparent 

2.9 

3.2 

2.6 

Other  relative 

1.3 

1.4 

1.1 

Nonrelative 

0.8 

1.0 

0.6 

Student  lives  with  (percent) 

Mother  (stepmother)  and  father  (stepfather) 

60.0 

58.5 

61.6 

Mother  (stepmother)  only;  father  (stepfather)  not  present 

33.6 

34.9 

32.3 

Father  (stepfather)  only;  mother  (stepmother)  not  present 

2.5 

2.3 

2.8 

Other  relative/nonrelative,  parents  not  present 

3.9 

4.4 

3.3 

Highest  education  of  anyone  in  household  (percent) 

Did  not  complete  high  school 

10.0 

8.7 

11.4 

Completed  high  school  or  equivalent 

21.9 

20.8 

23.0 

Some  college 

40.3 

42.4 

38.2 

Bachelor’s  or  higher  degree 

27.8 

28.1 

27.5 

Total  household  income  (percent) 

Less  than  $20,000 

33.2 

31.7 

34.7 

$20,000  to  $39,999 

24.4 

25.3 

23.4 

$40,000  to  $59,999 

15.1 

15.9 

14.4 

$60,000  or  more 

27.3 

27.1 

27.5 

Household  income-to-poverty  threshold  ratio  (percent) 

Below  135  percent 

39.0 

37.0 

41.0 

135  to  185  percent 

18.2 

18.9 

17.5 

Above  1 85  percent 

42.8 

44.1 

41.5 

Alabama  Parenting  Questionnaire — Poor  Monitoring  and 

Supervision  Subscale  (mean) 

1.2 

1.1 

1.2 

Alabama  Parenting  Questionnaire — Positive  Parenting 

Subscale  (mean) 

3.5 

3.5 

3.5 

Confusion,  Hubbub,  and  Order  Scale  (mean) 

2.2 

2.2 

2.2 

Community  characteristics  (mean) 

Community  Risks  Scale 

1.5 

1.5 

1.5 

Community  Resources  Scale 

2.7 

2.7 

2.7 

Child-Centered  Social  Control  Scale 

3.1 

3.1 

3.1 

* Treatment  group  significantly  different  from  control  group  at  the  .05  level. 

1 Categories  combined  to  protect  confidentiality. 

NOTE:  Weights,  which  assign  equal  weight  to  each  school  within  each  of  the  programs  and  to  each  program  across  programs,  were 
used  in  producing  the  treatment,  control,  and  overall  means.  Statistical  tests  were  conducted  using  regressions  that  included 
program  indicators  to  account  for  the  sample  design  and  adjusted  for  clustering  at  the  school  level. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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The  mean  values  of  the  outcome  measures  for  children’s  behavior  and  attitudes  as  reported  by  the  primary 
caregiver,  child,  and  teacher  in  fall  2004,  along  with  each  measure’s  range,  are  shown  in  table  1.13.  The  values 
were  similar  for  the  treatment  and  control  groups.  There  were  three  statistically  significant  differences  at  the 
program  level  (these  data  are  not  shown  in  a table)  but  their  number  and  pattern  suggest  that  they  may  have 
been  due  to  chance;  at  the  5 percent  significance  level,  6 of  the  126  comparisons  made  would  have  been 
expected  to  show  statistical  significance  by  chance.  Table  1.13  also  addresses  the  issue  of  whether  students 
had  room  to  improve  from  their  initial  scores  on  the  outcome  measures.  Five  of  the  mean  outcomes  were 
within  0.4  units  of  the  minimum  or  maximum  outcome  range  (Normative  Beliefs  About  Aggression; 
Altruistic  Behavior,  Teacher  Report  on  Student;  Problem  Behavior,  Child  Report  and  Teacher  Report  on 
Student;  and  Engagement  with  Learning).  However,  these  five  were  more  than  half  a standard  deviation  from 
their  minimum  or  maximum  value,  which  is  larger  than  the  minimal  detectable  impacts  for  the  study.  In 
addition,  the  other  mean  outcomes  were  one  or  more  standard  deviations  from  their  boundaries.  In  sum,  the 
evaluation  had  enough  power  to  identify  a minimum  detectable  effect  for  all  the  outcomes  before  potential 
ceiling  and  floor  effects  could  occur. 

The  mean  values  of  the  five  risk  factors  as  reported  in  fall  2004  showed  no  statistically  significant  difference 
between  the  treatment  and  control  groups  (table  1.14).  In  addition,  there  were  no  statistically  significant 
differences  at  the  program  level  (these  data  are  not  shown  in  a table) . 
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Table  1.13.  Mean  scores  and  standard  deviations  for  initial  outcome  measures 


Total  Treatment  Control 


Outcome  measure-Report 

Range 

Mean 

SD 

Mean 

SD 

Mean 

SD 

Social  and  Emotional  Competence  Domain 

Self-Efficacy  for  Peer  Interaction-CR 

1-4 

2.9 

0.6 

2.9 

0.6 

2.9 

0.6 

Normative  Beliefs  About  Aggression-CR 

1-4 

1.2 

0.4 

1.2 

0.4 

1.2 

0.4 

Empathy-CR 

1-3 

2.4 

0.4 

2.4 

0.4 

2.4 

0.4 

Behavior  Domain 

Altruistic  Behavior-CR 

0-3 

1.5 

0.8 

1.5 

0.9 

1.4 

0.8 

Altruistic  Behavior-TRS 

1-4 

1.4 

0.5 

1.4 

0.5 

1.4 

0.4 

Altruistic  Behavior-PCR 

1-4 

2.3 

0.7 

2.3 

0.7 

2.3 

0.7 

Positive  Social  Behavior-TRS 

1-4 

3.0 

0.7 

3.0 

0.7 

3.0 

0.7 

Positive  Social  Behavior-PCR 

1-4 

3.0 

0.5 

3.0 

0.5 

3.0 

0.5 

Problem  Behavior-CR 

0-3 

0.2 

0.4 

0.2 

0.4 

0.2 

0.4 

Problem  Behavior-TRS 

1-4 

1.4 

0.4 

1.4 

0.4 

1.3 

0.4 

Problem  Behavior-PCR 

1-4 

1.6 

0.3 

1.6 

0.3 

1.6 

0.3 

ADHD-Related  Behavior-TRS 

1-4 

1.7 

0.6 

1.7 

0.7 

1.7 

0.6 

Academics  Domain 

Academic  Competence  and  Motivation-TRS 

1-5 

2.9 

0.9 

2.9 

0.9 

2.9 

0.9 

Engagement  with  Learning-CR 

1-4 

3.7 

0.6 

3.7 

0.6 

3.7 

0.6 

Perceptions  of  School  Climate  Domain 

Positive  School  Orientation-CR 

1-4 

3.2 

0.7 

3.2 

0.7 

3.2 

0.6 

Negative  School  Orientation-CR 

1-4 

1.8 

0.7 

1.9 

0.7 

1.8 

0.7 

Student  Afraid  at  School-CR 

1-4 

2.4 

0.9 

2.4 

0.9 

2.4 

0.9 

Victimization  at  School-CR 

0-3 

0.8 

0.8 

0.8 

0.8 

0.8 

0.8 

Sample  size — PCR1 

3,774 

1,980 

1,794 

Sample  size — CR1 

3,997 

2,092 

1,905 

Sample  size — TRS1 

4,104 

2,158 

1,946 

1 Sample  size  may  differ  for  some  outcomes  due  to  nonresponse. 
NOTE:  Abbreviations  are 


CR:  Child  Report 

PCR:  Primary  Caregiver  Report 

TRS:  Teacher  Report  on  Student 

SD:  Standard  deviation 

ADHD:  Attention  deficit  hyperactivity  disorder 

No  findings  were  found  statistically  significant  at  or  below  the  .05  level.  Weights,  which  assign  equal  weight  to  each  school  within 
each  of  the  programs  and  to  each  program  across  programs,  were  used  in  producing  the  treatment,  control,  overall  means,  and 
standard  deviations.  Statistical  tests  were  conducted  using  regressions  that  included  program  indicators  to  account  for  the  sample 
design  and  adjusted  for  clustering  at  the  school  level. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Table  1.14.  Mean  scores  and  standard  deviations  for  initial  risk  factors 


Risk  factor 

Total 

Treatment 

Control 

Mean 

SD 

Mean 

SD 

Mean 

SD 

Family  risk 

1.2 

0.2 

1.1 

0.2 

1.2 

0.2 

Community  risk 

1.5 

0.7 

1.5 

0.7 

1.5 

0.7 

Child  behavior  risk-TRS 

50.9 

10.3 

51.2 

10.4 

50.5 

10.2 

Child  behavior  risk-PCR 

50.5 

10.1 

50.5 

9.8 

50.6 

10.3 

Socioeconomic  risk2 

No  risk  factors 

58.8 

49.2 

60.9 

48.8 

56.8 

49.5 

One  risk  factor 

29.8 

45.7 

29.0 

45.4 

30.6 

46.1 

Two  or  three  risk  factors 

11.4 

31.7 

10.1 

30.1 

12.6 

33.2 

Sample  size — PCR1 

3,774 

1,980 

1,794 

Sample  size — CR1 

3,997 

2,092 

1,905 

Sample  size — TRS1 

4,103 

2,157 

1,946 

1 Sample  size  may  differ  for  some  outcomes  due  to  nonresponse. 

2 As  a sensitivity  test,  the  index  was  also  treated  as  a continuous  variable  and  no  significant  differences  were  found. 
NOTE:  Abbreviations  are 


CR:  Child  Report 
PCR:  Primary  Caregiver  Report 
TRS:  Teacher  Report  on  Student 
SD:  Standard  deviation 

No  findings  were  found  statistically  significant  at  or  below  the  .05  level.  Weights,  which  assign  equal  weight  to  each  school  within 
each  of  the  programs  and  to  each  program  across  programs,  were  used  in  producing  the  treatment,  control,  overall  means,  and 
standard  deviations.  Statistical  tests  were  conducted  using  regressions  that  included  program  indicators  to  account  for  the  sample 
design  and  adjusted  for  clustering  at  the  school  level. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Characteristics  of  Teachers  and  Schools 

The  third-,  fourth-,  and  fifth-grade  teachers  at  the  study  schools  were  predominandy  White  non-Hispanic 
(75%),  female  (88%),  and  had  an  average  of  nearly  13  years  of  total  teaching  experience.  About  60  percent 
held  an  advanced  or  specialist  degree  (table  1.15).  Six  characteristics  were  compared,  with  less  than  one 
expected  to  be  significant  by  chance.  The  only  statistically  significant  difference  between  the  treatment  and 
control  group  of  teachers  concerned  degree  attainment:  a test  of  the  distribution  of  teachers’  education  levels 
across  five  categories  yielded  a />-value  of  0.043.  Among  the  programs  there  was  one  significant  difference 
versus  six  expected  by  chance:  for  PATHS,  male  teachers  made  up  a smaller  percentage  of  the  treatment 
teachers  than  the  control  teachers  (17%  versus  35%). 

Data  regarding  school  characteristics  were  drawn  from  the  U.S.  Department  of  Education’s  Common  Core 
of  Data  (2003-04)  in  order  to  compare  treatment  versus  control  schools  (table  1.16).  Comparisons  were  made 
on  nine  characteristics,  with  less  than  one  expected  to  be  found  significant  by  chance.  These  characteristics 
included  student  composition  (race/ethnicity  and  school  lunch  eligibility),  number  of  students,  number  of 
full-time  teachers,  Title  I status,  highest  and  lowest  grades,  school  location,  and  years  the  principal  had  been 
at  the  school.  Overall,  one  significant  difference  was  found:  a smaller  percentage  of  treatment  schools  than 
control  schools  (50%  versus  67%)  had  prekindergarten.  No  significant  differences  were  found  among  the 
programs. 

In  the  Teacher  Report  on  Classroom  and  School,  teachers  reported  on  nine  dimensions  concerning  their 
school  environment,  including  feelings  of  safety,  resource  adequacy,  student  support,  staff  freedom  to  teach 
as  desired,  affiliation  with  and  ties  to  colleagues,  innovation  regarding  trying  new  approaches  to  teaching, 
professional  interest,  participatory  decisionmaking,  and  work  pressure.  There  were  no  statistically  significant 
differences  (these  data  are  not  shown  in  a table)  in  these  reports  between  treatment  and  control  schools, 
overall  or  at  the  program  level. 
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Table  1.15.  Initial  characteristics  of  teachers 


Characteristic 

Total 

Treatment 

Control 

Teacher  sample  size 

847 

436 

411 

Gender  (percent) 

Male 

12.3 

11.9 

12.7 

Female 

87.7 

88.1 

87.3 

Race/ethnicity  (percent) 

White  (non-Hispanic) 

75.2 

73.1 

77.4 

Black  (non-Hispanic) 

16.8 

18.6 

15.1 

Hispanic 

4.9 

5.9 

3.8 

Other 

3.1 

2.4 

3.7 

Number  of  years  teaching  (mean) 

12.8 

12.5 

13.0 

Number  of  years  teaching  in  this  school  (mean) 

7.3 

6.8 

7.7 

Type  of  teaching  certificate  (percent) 

Regular  state  certificate  or  advanced  professional  certificate 

84.7 

83.4 

86.0 

Provisional  certificate  (for  those  in  alternative  certification  programs) 

6.2 

6.8 

5.5 

Probationary  certificate  (for  those  who  have  satisfied  all 
requirements  except  for  completing  the  probationary  period) 

4.9 

5.7 

4.2 

Emergency  certificate  or  waiver  (for  those  who  must 
complete  a certification  program  to  continue  teaching) 

0.0 

0.0 

0.0 

Other1 

4.1 

4.1 

4.2 

Education  (percent) 

.2 

Less  than  a bachelor’s  degree 

0.0 

0.0 

0.0 

Bachelor’s  degree 

40.2 

37.6 

42.8 

Master’s  degree,  Ph.D. 

56.2 

60.2 

52.3 

Specialist  degree 

2.1 

1.0 

3.2 

Other 

1.5 

1.3 

1.7 

* Treatment  group  significantly  different  from  control  group  at  the  .05  level. 

1 Categories,  including  no  certificate  and  temporary  certificate,  combined  to  protect  confidentiality. 

2 A test  of  the  distribution  of  teachers’  education  levels  across  all  five  categories  yielded  a p-value  of  0.043. 

NOTE:  Weights,  which  assign  equal  weight  to  each  school  within  each  of  the  programs  and  to  each  program  across  programs,  were 
used  in  producing  the  treatment,  control,  and  overall  means.  Statistical  tests  were  conducted  using  regressions  that  included 
program  indicators  to  account  for  the  sample  design  and  adjusted  for  clustering  at  the  school  level. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Table  1.16.  Initial  characteristics  of  schools 


Characteristic 

Total 

Treatment 

Control 

School  sample  size 

84 

42 

42 

Student  race/ethnicity  (percent) 

White  (non-Hispanic) 

37.3 

36.4 

38.3 

Black  (non-Hispanic) 

39.7 

40.9 

38.5 

Hispanic 

18.7 

18.0 

19.5 

Other 

4.2 

4.6 

3.8 

Students  eligible  for  free  or  reduced-price  lunch  (percent) 

61.3 

62.3 

60.4 

Number  of  students  enrolled  (mean) 

567.1 

555.7 

578.9 

Number  of  full-time  teachers  (mean) 

38.5 

37.3 

39.6 

Title  1 status  (percent) 

Title  1 eligible  school 

74.2 

73.2 

75.2 

Schoolwide  Title  1 

63.7 

61.2 

66.2 

Lowest  grade  offered  (percent) 

Prekindergarten 

58.5 

50.4* 

66.7 

Kindergarten 

41.5 

49.6 

33.3 

Highest  grade  offered  (percent) 

Fifth  grade 

55.4 

58.1 

52.8 

Sixth  grade 

22.0 

19.0 

25.1 

Eighth  grade 

22.5 

22.9 

22.1 

Location  of  school  (percent) 

City 

56.2 

58.6 

53.8 

Suburbs 

26.7 

25.2 

28.1 

Rural 

17.1 

16.2 

18.1 

Number  of  years  principal  has  been  at  this  school  (mean) 

5.5 

4.8 

6.1 

* Treatment  group  significantly  different  from  control  group  at  the  .05  level. 

NOTE:  Weights,  which  assign  equal  weight  to  each  school  within  each  of  the  programs  and  to  each  program  across  programs,  were 
used  in  producing  the  treatment,  control,  and  overall  means.  Statistical  tests  were  conducted  using  regressions  that  included 
program  indicators  to  account  for  the  sample  design  and  adjusted  for  clustering  at  the  school  level.  Data  are  missing  from  Love  In  a 
Big  World  for  student  race/ethnicity,  percentage  of  students  eligible  for  free  or  reduced-price  lunch,  and  number  of  full-time  teachers. 
SOURCE:  NCES  Common  Core  of  Data  (2003-04),  The  Social  and  Character  Development  (SACD)  Research  Program. 
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The  Initial  Level  of  Social  and  Character  Development  Activities  in  the  Schools 

During  the  initial  data  collection  in  fall  2004,  principals  and  teachers  reported  on  the  SACD  activities  used  in 
the  schools  and  classrooms,  the  availability  of  SACD  materials  and  instructional  practices  used,  and  the 
professional  development  teachers  received  on  SACD  over  the  past  12  months.  These  reports  were  likely  to 
have  been  influenced  by  the  implementation  of  six  programs  (in  36  of  the  42  treatment  schools)  and  the 
training  of  treatment  teachers  in  all  schools  that  preceded  the  data  collection.  For  this  reason,  the  fall  2004 
data  did  not  provide  baseline  data  on  how  the  use  of  SACD  activities  differed  between  treatment  and  control 
before  the  intervention  but,  instead,  provided  data  on  how  this  use  differed  early  on  in  the  study. 

Table  1.17  shows  that  most  school  principals  reported  activities  to  promote  six  social  and  character 
development  goals:  violence  prevention  and  peace  promotion  (93%),  social  and  emotional  development 
(95%),  character  education  (96%),  tolerance  and  diversity  (86%),  risk  prevention  and  health  promotion  (81%), 
and  civic  responsibility  and  community  service  (90%).  In  addition,  most  (91%)  also  reported  activities 
directed  toward  behavior  management.  There  were  no  statistically  significant  differences  in  the  percentages  of 
principals  who  reported,  in  the  treatment  group  schools  and  the  control  group  schools,  that  their  school  had 
programs  or  activities  to  promote  each  of  these  goals. 

Most  teachers  reported  the  use  of  these  activities  in  their  classroom  to  promote  the  six  social  and  character 
development  goals  and  behavior  management  (ranging  from  53%  to  87%).  From  the  eight  comparisons  made 
(including  a “none  of  the  above”  comparison)  less  than  one  significant  difference  would  have  been  expected. 
Unlike  the  principals,  a significantly  larger  percentage  of  treatment  teachers  than  control  teachers  reported 
using  activities  to  promote  three  SACD  goals:  violence  prevention  and  peace  promotion  activities  (70% 
versus  59%),  social  and  emotional  development  activities  (79%  versus  68%),  and  behavior  management  (90% 
versus  84%).  In  addition,  seven  comparisons  were  made  as  to  whether  teachers  used  the  activities  for  each  of 
the  six  SACD  goals  and  behavior  management  with  all,  some,  or  no  students  (less  than  one  significant 
difference  would  be  expected  by  chance).  Treatment  teachers  were  statistically  significantly  more  likely  to 
report  engaging  all  students,  as  opposed  to  some  or  no  students,  with  activities  to  promote  the  same  three 
SACD  goals  (these  data  are  not  shown  in  a table). 

The  majority  of  teachers  reported  that  their  schools  used  schoolwide  activities  to  promote  social  and 
character  development.  These  activities  included  morning  announcements  or  videos  (reported  by  80%  of 
teachers),  school  assemblies  (71%),  school  newspapers  or  bulletins  (67%),  special  school  days  (54%),  and 
special  events  (72%).  Only  13  percent  reported  the  use  of  other  activities.  There  were  no  significant 
differences  in  the  percent  of  treatment  versus  control  teachers  reporting  the  use  of  these  six  activities. 


65 


Chapter  1 . The  Social  and  Character  Development  Multiprogram  Evaluation 


Table  1.17.  Principal  and  teacher  initial  reports  on  use  of  SACD  programs  or  activities 


SACD  program  or  activity 

Total 

Treatment 

Control 

Principal  sample  size 

84 

42 

42 

Teacher  sample  size 

847 

436 

411 

Principals  reporting  that  school  had  programs  or  activities 
to  promote  the  following  SACD  goals  (percent) 

Violence  prevention  and  peace  promotion 

92.8 

92.9 

92.7 

Social  and  emotional  development 

95.0 

95.3 

94.7 

Character  education 

96.4 

95.6 

97.1 

Tolerance  and  diversity 

85.8 

84.7 

87.0 

Risk  prevention  and  health  promotion 

80.8 

85.1 

76.5 

Civic  responsibility  and  community  service 

89.7 

92.3 

87.0 

Behavior  management 

91.4 

93.9 

89.0 

Teachers  reporting  on  using  programs  or  activities  in  their 
class  to  promote  the  following  SACD  goals  (percent) 

Violence  prevention  and  peace  promotion 

64.6 

69.8* 

59.4 

Social  and  emotional  development 

73.2 

78.5** 

67.9 

Character  education 

82.5 

84.4 

80.5 

Tolerance  and  diversity 

63.3 

64.4 

62.1 

Risk  prevention  and  health  promotion 

52.9 

51.7 

54.1 

Civic  responsibility  and  community  service 

59.2 

59.6 

58.7 

Behavior  management 

86.8 

89.6* 

83.9 

None  of  the  above 

2.4 

1.6 

3.2 

Teachers  reporting  schoolwide  use  of  the  following 
activities  to  promote  SACD  (percent) 

Morning  announcements  or  videos 

80.2 

77.2 

83.3 

School  assemblies 

71.0 

69.1 

72.9 

School  newspapers  or  bulletins 

66.7 

62.3 

71.1 

Special  school  days 

53.8 

53.8 

53.9 

Special  events 

71.5 

72.5 

70.5 

Other  activities 

12.8 

12.5 

13.1 

* Treatment  group  significantly  different  from  control  group  at  the  .05  level. 


**  Treatment  group  significantly  different  from  control  group  at  the  .01  level. 

NOTE:  Weights,  which  assign  equal  weight  to  each  school  within  each  of  the  programs  and  to  each  program  across  programs,  were 
used  in  producing  the  treatment,  control,  and  overall  means.  Statistical  tests  were  conducted  using  regressions  that  included 
program  indicators  to  account  for  the  sample  design  and  adjusted  for  clustering  at  the  school  level.  Sample  size  may  differ  for  some 
outcomes  due  to  nonresponse. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Teachers  reported  on  their  use  of  a broad  range  of  teaching  materials  (table  1.18).  The  materials  most  used 
with  SACD  activities  were  teacher  guides  (used  by  65%  of  teachers),  student  materials  (used  by  50%),  and 
children’s  literature  (used  by  50%).  Of  the  seven  materials  reported  on,  teachers  at  treatment  schools  had 
three  reports  of  use  that  statistically  significandy  differed  from  teachers  in  control  schools  (versus  less  than 
one  by  chance):  teacher  guides  (72%  versus  59%),  instructional  aids  such  as  games  or  videos  (37%  versus 
29%),  and  other  types  of  materials  (11%  versus  16%).  There  were  no  significant  differences  in  the  reported 
use  of  student  materials,  children’s  literature,  giveaway  materials,  or  in  not  using  any  of  these  materials. 

Teachers  reported  using  a wide  variety  of  instructional  strategies  (table  1.18).  Treatment  teachers  reported 
significantly  higher  use  on  5 of  the  22  items  than  control  teachers  (1  significant  difference  would  have  been 
expected  by  chance).  These  included  role-playing  (73%  versus  59%),  direct  instruction  of  social  and  character 
development  (87%  versus  76%),  incorporating  social  and  character  development  into  the  academic 
curriculum  (80%  versus  69%),  targeted  story  reading  or  writing  on  social  and  character  development  themes 
(80%  versus  71%),  and  the  average  number  of  strategies  used  (12.2  versus  11.3). 
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Table  1.18.  Teacher  initial  reports  on  use  of  SACD  materials  and  classroom  strategies 


SACD  material  and  strategy 

Total 

Treatment 

Control 

Teacher  sample  size 

847 

436 

411 

Teachers  using  the  following  materials  in  conjunction  with 
social  and  character  development  activities  (percent) 

Teacher  guides  (manuals,  curricula) 

65.3 

71.6** 

59.1 

Student  materials  (workbooks,  worksheets) 

50.1 

53.1 

47.1 

Instructional  aids  (games,  software,  videos) 

33.2 

37.4* 

28.9 

Giveaways  (bookmarks,  stickers) 

47.2 

45.3 

49.1 

Children’s  literature 

50.0 

53.5 

46.6 

Other  types  of  materials 

13.2 

10.6* 

15.8 

Do  not  use  any  of  the  materials  listed  above 

10.1 

9.1 

11.1 

Teachers  using  any  of  the  strategies  listed  below  to  promote 
social  and  character  development  in  the  classroom  (percent) 

99.8 

99.6 

100.0 

Number  of  strategies  (listed  below)  used  by  teachers  to  promote 
social  and  character  development  in  the  classroom  (mean) 

11.8 

12.2** 

11.3 

Teachers  using  each  of  the  following  strategies  to  promote 
social  and  character  development  (percent) 

Role-playing 

65.7 

72.5** 

58.9 

Cooperative  learning 

96.1 

97.0 

95.3 

Peer  group  discussions 

87.7 

90.3 

85.1 

Direct  instruction  of  social  and  character  development 

81.6 

87.4** 

75.9 

Skill  training 

43.1 

46.8 

39.5 

Incorporating  social  and  character  development  into 
academic  curriculum 

74.7 

80.3** 

69.2 

Parent  training 

7.8 

6.8 

8.9 

Parent/community  involvement  in  program  development 
or  delivery 

24.0 

26.7 

21.4 

Mentoring 

36.3 

35.8 

36.7 

Good  behavior  notes  sent  home  daily  or  weekly 

75.4 

76.7 

74.1 

Presenting  role  models 

67.2 

68.2 

66.2 

Targeted  story  reading  or  writing  on  SACD  themes 

75.1 

79.8* 

70.5 

Peer  mediation 

38.8 

40.9 

36.7 

Honor  roll  for  positive  behavior 

51.2 

50.4 

52.1 

Pledges  or  recitations  on  social  and  character 
development  themes 

38.1 

38.6 

37.6 

Guided  visualization 

44.9 

47.2 

42.7 

Student-led/student-assisted  instruction 

48.5 

50.9 

46.1 

Journaling 

72.0 

73.6 

70.3 

Time  out  for  negative  behavior 

84.9 

86.5 

83.3 

Daily  or  weekly  rewards  for  positive  behavior 

92.0 

91.6 

92.3 

* Treatment  group  significantly  different  from  control  group  at  the  .05  level. 

**  Treatment  group  significantly  different  from  control  group  at  the  .01  level. 

NOTE:  Weights,  which  assign  equal  weight  to  each  school  within  each  of  the  programs  and  to  each  program  across  programs,  were 
used  in  producing  the  treatment,  control,  and  overall  means.  Statistical  tests  were  conducted  using  regressions  that  included 
program  indicators  to  account  for  the  sample  design  and  adjusted  for  clustering  at  the  school  level.  Sample  size  may  differ  for  some 
outcomes  due  to  nonresponse. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Principals  and  teachers  reported  on  participation  in  and  amount  of  SACD  training  and  staff  development 
provided  over  the  previous  12  months  (table  1.19).  This  period  overlapped  with  the  time  that  treatment 
schools  received  training  in  the  SACD  programs.  Principals  reported  staff  participation  rates  of  92  percent 
and  a mean  of  8.3  hours  of  training,  with  no  significant  difference  between  treatment  and  control  principals. 

On  the  nine  training  items  asked  of  teachers,  treatment  teachers’  reports  were  statistically  significandy  higher 
for  five  of  them  (versus  less  than  one  expected  by  chance).  These  included  staff  participation  rates  in  SACD 
training  over  the  past  12  months  (86%  for  treatment  teachers  versus  63%  for  control  teachers),  number  of 
training  hours  received  (9.5  versus  6.4),  and  participation  rates  in  training  in  specific  areas:  social  and 
emotional  development  (39%  versus  17%),  character  education  (55%  versus  27%),  and  behavior  management 
(42%  versus  32%). 


Table  1.19.  Principal  and  teacher  initial  reports  on  SACD  professional  development 


SACD  professional  development 

Total 

Treatment 

Control 

Principal  sample  size 

84 

42 

42 

Teacher  sample  size 

847 

436 

411 

Principals  reporting  that  staff  participated  in  social  and  character 
development  training  within  the  past  year  (percent) 

92.0 

97.5 

86.2 

Teachers  reporting  participation  in  social  and  character 
development  training  within  the  past  12  months  (percent) 

74.3 

86.1** 

62.5 

Number  of  hours  of  social  and  character  development  training 
principals  report  were  provided  to  each  staff  person  last  year  (mean) 

8.3 

9.4 

7.1 

Number  of  hours  of  social  and  character  development  training 
teachers  report  receiving  during  the  past  12  months  (mean) 

7.9 

9.5** 

6.4 

Teachers  reporting  receiving  training  in  the  past  12  months 
in  the  following  areas  (percent) 

Violence  prevention  and  peace  promotion 

22.6 

23.8 

21.5 

Social  and  emotional  development 

27.9 

38.5** 

17.2 

Character  education 

40.9 

54.8** 

26.9 

Tolerance  and  diversity 

18.8 

20.8 

16.9 

Risk  prevention  and  health  promotion 

11.3 

11.5 

11.1 

Civic  responsibility  and  community  service 

6.6 

6.4 

6.7 

Behavior  management 

36.8 

41.6* 

32.0 

* Treatment  group  significantly  different  from  control  group  at  the  .05  level. 


**  Treatment  group  significantly  different  from  control  group  at  the  .01  level. 

NOTE:  Weights,  which  assign  equal  weight  to  each  school  within  each  of  the  programs  and  to  each  program  across  programs,  were 
used  in  producing  the  treatment,  control,  and  overall  means.  Statistical  tests  were  conducted  using  regressions  that  included 
program  indicators  to  account  for  the  sample  design  and  adjusted  for  clustering  at  the  school  level.  Sample  size  may  differ  for  some 
outcomes  due  to  nonresponse. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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The  data  on  the  initial  level  of  SACD  activity  led  to  two  findings.  First,  treatment  teachers  reported  greater 
use  of  and  training  in  SACD  activities  than  control  teachers  more  often  than  would  be  expected  by  chance. 
There  are  two  potential  causes  for  this  finding,  and  the  analysis  cannot  be  used  to  determine  whether  the 
reason  for  such  a difference  was  that  the  two  groups  did  differ  on  their  initial  use  of  SACD  activities  (i.e.,  that 
randomization  did  not  create  similar  treatment  and  control  groups)  or  that  the  training  of  all  treatment 
teachers  and  the  implementation  of  six  of  the  programs  before  the  initial  data  were  collected  influenced  the 
teacher  reports.  Because  it  is  likely  (though  unproven)  that  the  training  and  implementation  affected  the 
teacher  reports,  these  data  were  not  considered  appropriate  for  use  as  a baseline  measure  of  SACD  activities 
and  training  in  the  treatment  schools. 

Second,  these  data  indicate  that  the  control  condition  for  the  SACD  project  was  not  a “no  treatment”  control 
but  a “standard  practice”  control.  Because  the  control  teachers  were  not  affected  by  the  implementation  of 
the  SACD  programs  before  data  collection,  their  reports  reflect  standard  practice  in  the  control  schools. 
Standard  practice  at  the  control  schools  included  reports  of  54  percent  to  84  percent  of  teachers  using  SACD 
activities,  89  percent  of  teachers  reporting  the  use  of  specific  materials  in  conjunction  with  these  activities, 
100  percent  reporting  the  use  of  at  least  one  of  the  specified  instructional  strategies,  and  63  percent  reporting 
participation  in  SACD  training  over  the  past  12  months. 

Year-by-Year  Impacts  on  Use  of  Social  and  Character  Development 
Activities 

Under  the  first  research  questions,  the  SACD  programs  were  expected  to  increase  the  use  of  SACD  activities 
in  the  treatment  schools  in  comparison  to  the  control  schools.  Two  analyses  of  this  impact  were  done:  one 
based  on  teacher  reports  and  the  other  on  principal  reports. 

Analysis  of  Teacher  Reports 

Every  spring,  third-,  fourth-,  and  fifth -grade  teachers  provided  information  through  the  Teacher  Report  on 
Classroom  and  School  about  the  social  and  character  development  activities  they  used  in  their  classroom.  As 
described  in  the  Measures  section,  83  variables  were  derived  from  individual  items  on  the  Teacher  Report  on 
Classroom  and  School.  These  were  grouped  under  six  domains  and  used  to  determine  the  difference  between 
treatment  and  control  teachers: 

1.  The  use  of  SACD  activities  in  their  classrooms  to  promote  the  six  social  and  character  development 
goals  (separately  and  all  together)  and  behavior  management,  and  the  use  of  such  activities  for  at  least 
1 hour  per  week  for  the  same  purposes; 

2.  The  use  of  SACD  activities  linked  to  named  SACD  programs  in  their  classrooms  to  promote  the  six 
social  and  character  development  goals  (separately  and  all  together),  and  the  use  of  such  activities  for 
at  least  1 hour  per  week  for  the  same  purposes; 

3.  The  use  of  materials  and  strategies  to  implement  the  activities  in  the  classroom; 

4.  The  use  of  six  schoolwide  strategies  to  promote  social  and  character  development; 

5.  The  use  of  staff  development  to  support  the  teachers;  and 

6.  Staff  attitudes  toward  SACD  activities  and  the  use  of  practices  conducive  to  the  development  of 
social  and  character  development. 

Teacher  Report  on  Classroom  and  School  consent  and  completion  rates  (presented  in  table  1.9)  led  to  87 
percent  to  90  percent  of  all  teachers  having  data  for  the  3 years  (varying  by  1%  to  2%  for  treatment  versus 
control  teachers).  At  the  program  level,  the  data  available  ranged  from  83  percent  to  100  percent  of  all 
teachers  for  the  3 years. 
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To  estimate  the  impacts  of  the  SACD  programs  for  each  of  the  outcome  measures,  testing  the  statistical 
significance  of  the  differences  in  means  between  the  treatment  and  the  control  teachers  was  used.22  Before 
differences  in  the  means  were  tested,  the  data  were  weighted  such  that  each  school  received  equal  weight 
within  a program  and  each  program  received  equal  weight  in  the  combined -program  estimates.  Standard 
errors  of  the  impact  estimates  accounted  both  for  unequal  weighting  and  for  the  clustering  of  teachers  within 
schools. 

In  addition  to  estimating  the  impacts  of  the  SACD  programs  on  the  individual  outcome  measures,  the 
impacts  on  the  six  domains  were  also  examined.  Testing  the  impact  on  the  domains  was  done  to  adjust  for 
the  multiple  comparisons  made  within  each  domain  in  order  to  address  the  increased  chances  of  finding  a 
spurious  outcome  when  more  than  one  test  was  done.  As  a result,  this  section  provides  two  sets  of  results: 
(1)  the  impacts  on  the  individual  outcomes  unadjusted  for  multiple  comparisons,  and  (2)  the  impacts  on  the 
domains  that  serve  as  the  multiple  comparison  adjustment.  23 

The  testing  of  the  significance  of  the  impacts  on  the  domains  was  based  on  a set  of  three  heuristics  modeled 
on  the  approach  used  by  the  U.S.  Department  of  Education’s  What  Works  Clearinghouse  for  determining 
whether  domain-level  effects  are  statistically  significant  when  there  are  multiple  outcome  measures  within  a 
single  domain  (What  Works  Clearinghouse  n.d.b).  Each  domain  was  checked  using  the  three  heuristics,  and  a 
statistically  significant  positive  impact  was  found  if  any  of  the  three  were  met.24  The  three  heuristics  were  as 
follows: 

1.  Based  on  the  results  from  the  statistical  test  of  each  outcome  variable  within  a domain  (unadjusted 
for  multiple  comparisons),  at  least  half  of  the  impacts  were  positive  and  statistically  significant  and  no 
impact  was  negative  and  statistically  significant. 

2.  The  omnibus  impact  for  all  the  outcomes  measured  together  was  positive  and  statistically  significant 
on  the  basis  of  a multivariate  statistical  test. 25 


22  Three  factors  contributed  to  the  decision  to  use  differences  in  means.  First,  because  of  random  assignment,  simple 
treatment-control  contrasts  provided  unbiased  estimates  of  program  impacts.  Second,  only  initial  values  (rather  than  true 
baseline  values)  for  these  outcomes  were  available  to  use  in  a model  because  training  (at  all  treatment  schools)  and 
program  implementation  (at  36  treatment  schools)  began  before  data  collection.  The  decision  to  use  initial  values  in  an 
analysis  partly  depends  on  whether  the  initial  training  and  implementation  occurring  before  data  collection  would  be 
expected  to  have  immediate  and  large  impacts  on  the  outcomes  (Schochet  2008b).  For  this  analysis,  the  outcomes  are 
based  on  teacher  actions  and  so  would  likely  be  upwardly  influenced  by  the  teacher  training  and  short  period  of  teacher 
implementation  before  pretesting  (in  contrast  to  student  outcomes,  which  would  be  less  likely  to  be  so  influenced).  For 
this  reason  a model-based  analysis  using  the  initial  values  as  covariates  was  not  chosen.  Third,  preliminary  analyses 
indicated  no  gain  in  precision  from  the  inclusion  of  other  covariates. 

23  When  the  SACD  evaluation  was  designed,  there  was  no  general  agreement  in  the  field  about  how  best  to  adjust  for 
multiple  comparisons,  and  this  approach  was  seen  by  the  SACD  Research  Consortium  as  the  best  method  available.  For 
a more  recent  discussion  of  the  issue,  see  Schochet  2008a. 

24  For  ease  of  discussion,  this  section  describes  detecting  positive  significant  impacts.  These  heuristics  were  also  used  to 
detect  a significant  negative  impact  on  a domain. 

25  A multivariate  analysis  of  variance  (MANOVA)  procedure  (Flarris  1975)  was  used.  Suppose  there  are  p domain 
outcomes  (the  dependent  variables),  n sample  members,  and  for  each  dependent  variable  there  is  one  treatment  effect 
parameter  and  k parameters  for  other  covariates.  The  MANOVA  model  can  then  be  expressed  as  follows: 

Y- Ta  + Xb  + e 

where  Y is  a >ixp  matrix  of  domain  outcomes,  T is  an  n x 1 vector  of  treatment  indicators,  a is  a / mp  parameter  vector  of 
treatment  effects,  X is  an  n x k matrix  of  covariates  (a  limited  set  of  covariates  that  included  indicator  variables  for  the 
programs  and  the  intervention)  with  associated  parameter  matrix  b,  and  e is  a n xp  matrix  of  error  terms  that  accounts 
for  school-level  clustering.  The  omnibus  F-test  uses  the  MANOVA  model  to  test  the  joint  null  hypothesis  of  zero 
treatment  effects  for  each  domain  outcome  (that  is,  Flo:  a — 0).  Wilks’  lambda  was  used  to  assess  statistical  significance. 
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3.  After  applying  the  Benjamini-Hochberg  (Benjamini  and  Hochberg  1995)  procedure  to  each  outcome 
under  a domain,  at  least  one  of  the  outcomes  remained  positive  and  statistically  significant  and  no 
outcome  was  negative  and  statistically  significant.  The  Benjamini-Hochberg  procedure  adjusts 
significance  levels  downward  to  account  for  the  multiple  testing  of  impacts. 

A statistically  significant  impact  on  a domain  was  recognized  if  one  or  more  of  the  three  heuristics  was  met. 
There  could  be  situations  in  which  one  or  more  of  the  heuristics  indicated  a statistically  significant  impact  of 
the  intervention  on  the  domain,  when  the  preponderance  (or  all)  of  the  individual  outcomes  did  not  indicate 
statistically  significant  impacts.  For  example,  this  type  of  situation  might  have  arisen  when  a test  captured  a 
latent  construct  underlying  the  individual  outcomes  within  the  domain;  the  latent  construct  might  have  had 
less  statistical  noise  than  did  the  individual  outcomes  and  thus  been  more  likely  to  have  indicated  statistical 
significance.  There  also  could  be  situations  in  which  one  or  more  of  the  individual  outcomes  showed 
statistically  significant  impacts  but  none  of  the  heuristics  showed  a significant  impact  on  the  domain. 

Analysis  of  Principal  Interviews 

Every  spring  principals  were  individually  interviewed  about  the  prevalence  and  characteristics  of  the  activities 
used  to  promote  social  and  character  development  along  with  the  percentage  of  staff  trained  in  them  and 
staff  support  of  their  use.  As  discussed  in  the  Measures  section,  information  from  the  interviews  could  be 
used  to  determine  the  difference  in  reports  between  treatment  and  control  principals  regarding  (1)  the 
percentages  of  schools  using  activities  to  address  each  of  the  six  SACD  goals,  (2)  the  delivery  methods  for 
these  SACD  activities,  (3)  targeting  of  the  activities,  (4)  teacher  professional  development,  and  (5)  staff 
support  for  the  promotion  of  social  and  character  development. 

The  interviews  had  an  almost  perfect  response  rate;  all  principals  completed  them  in  Years  1 and  2 and  all  but 
one  control  school  principal  completed  them  in  Year  3.  Because  of  inconsistencies  in  the  quality  of 
interviewers’  recording  of  answers  to  the  open-ended  questions  during  the  spring  2005  interviews,  the  results 
from  the  spring  2005  open-ended  data  are  not  included. 

The  interview  data  on  activity  level  for  the  six  SACD  goals  were  transformed  in  two  ways:  (1)  an  indicator  of 
whether  the  school  had  any  activities  that  addressed  each  SACD  goal  was  created,  and  (2)  the  number  of 
activities  in  a school  that  addressed  each  SACD  goal  was  calculated.  Logistic  regression  was  used  to 
determine  whether  there  were  differences  between  treatment  and  control  schools  in  the  likelihood  of  each 
implementation  measure  that  was  measured  through  a dichotomous  school-level  variable.  Ordinary  least 
squares  regression  was  used  to  determine  whether  there  were  differences  between  the  treatment  and  control 
schools  in  the  number  of  activities  to  promote  SACD  goals  within  schools.  The  research  team-specific 
characteristics  were  controlled  for  in  the  model  by  including  indicator  variables  for  each  team.  An  indicator 
variable  of  school-level  treatment  status  was  included  in  the  model.  No  other  covariates  were  used  in  the 
models  because  of  the  small  sample  size.  The  What  Works  Clearinghouse  heuristics  were  not  used  in  this 
analysis,  and  the  analysis  was  not  done  for  each  individual  SACD  program  because  of  the  small  sample  sizes. 

Results  From  Teacher  Reports 

The  SACD  programs  were  expected  to  increase  the  use  of  SACD  activities  in  the  classroom  and  school, 
broaden  the  types  of  materials  and  instructional  strategies  used  to  implement  these  activities  in  the  classroom 
and  schoolwide,  increase  the  amount  of  staff  development  for  such  activities,  and  generate  positive  attitudes 
and  schoolwide  practices.  Eighty-three  outcomes  were  measured  each  year  for  a total  of  249  comparisons 
between  treatment  and  control  teachers  tested  over  the  3 years  (with  12  to  13  expected  to  be  found 
statistically  significant  by  chance).  The  analysis  found  127  statistically  significant  differences  in  these  249 
comparisons,  with  all  showing  greater  reported  use  of  SACD  activities  by  treatment  teachers.  Here,  the 
impacts  on  the  outcomes  (organized  by  domain)  are  described  and  followed  by  a discussion  of  the  analysis  of 
the  impacts  on  the  domains. 
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Use  of  Activities  and  Activities  Linked  to  Named  SACD  Programs 

Two  patterns  of  findings  emerged  from  a comparison  of  treatment-control  group  differences  in  teacher- 
reported  activities  to  promote  social  and  character  development  goals  in  the  classroom  and  the  use  of 
behavior  management.  First,  treatment  teachers  were  more  likely  than  control  teachers  to  report 
implementing  these  activities  in  their  classrooms  and  more  likely  to  report  conducting  these  activities  for  at 
least  1 hour  per  week  in  general  and  when  reporting  on  activities  linked  to  named  SACD  programs.  Second, 
control  group  teachers  also  reported  conducting  activities  to  promote  social  and  character  development. 

Table  1.20  shows  the  percentages  of  teachers  who  reported  activities  to  promote  each  of  six  social  and 
character  development  goals,  the  percentages  who  reported  activities  to  promote  any  one  of  the  six  goals,  and 
the  percentages  who  reported  that  they  used  behavior  management  techniques.  Impact  estimates  (which 
represent  the  difference  in  percentile  points  between  treatment  and  control  teachers  who  reported  positively) 
are  presented  for  each  school  year.  The  table  is  divided  into  four  panels:  (1)  engagement  in  any  activity, 
(2)  engagement  in  any  activity  for  at  least  an  hour  per  week  on  average,  (3)  engagement  in  any  activity  as  part 
of  a named  program,  and  (4)  engagement  in  any  activity  as  part  of  a named  program  for  at  least  an  hour  a 
week. 

Measuring  teacher  activity  in  these  diverse  ways  contributes  to  an  understanding  of  all  activities  that  a teacher 
considered  as  promoting  one  of  the  six  social  and  character  development  goals,  distinguishes  between 
activities  that  were  conducted  intensively  (defined  as  at  least  1 hour  per  week)26  from  those  that  might  have 
been  done  only  occasionally,  and  distinguishes  between  activities  that  were  conducted  as  part  of  a named 
program  and  activities  that  might  have  been  less  structured  and  had  less  institutional  support. 

The  first  pattern  of  a larger  percentage  of  treatment  teachers  than  control  teachers  reporting  engagement  in 
SACD  activities  can  be  seen  throughout  table  1.20.  In  regard  to  teachers’  reported  use  of  SACD  activities 
(panels  1 and  2),  48  comparisons  were  made  over  the  3 years,  with  3 expected  to  be  significant  by  chance.  A 
significantly  greater  percentage  of  treatment  teachers  reported  using  SACD  activities  than  control  teachers  in 
31  of  these  48  comparisons.  In  regard  to  teachers’  reported  use  of  SACD  activities  linked  to  named  SACD 
programs  (panels  3 and  4),  42  comparisons  were  made  over  the  3 years  with  2 expected  to  be  significant  by 
chance.  A significantly  greater  percentage  of  treatment  teachers  reported  using  the  SACD  activities  in  39  of 
these  42  comparisons.  In  all  3 years,  the  percentages  of  treatment  teachers  reporting  engagement  in  SACD 
activities  were  statistically  significantly  greater  than  for  control  teachers  in  regard  to  promoting  any  of  the 
SACD  goals  and  promoting  four  of  the  six  individual  SACD  goals  (violence  prevention  and  peace  promotion, 
social  and  emotional  development,  character  education,  and  tolerance  and  diversity).  For  example,  when 
looking  at  activities  to  support  any  SACD  goal,  the  statistically  significant  difference  favoring  treatment 
teachers  ranged  from  5 to  10  percentage  points  for  engagement  in  any  activities  (panel  1,  Any  SACD  goal),  7 
to  24  percentage  points  for  engagement  in  any  activity  for  at  least  an  hour  per  week  (panel  2,  Any  SACD 
goal),  29  to  34  percentage  points  for  engagement  in  any  activity  linked  to  a named  SACD  program  (panel  3, 
Any  SACD  goal),  and  26  to  34  percentage  points  for  engagement  in  any  activity  linked  to  a named  SACD 
program  for  at  least  an  hour  per  week  (panel  4,  Any  SACD  goal). 

The  second  pattern  of  control  teachers  reporting  engagement  in  SACD  activities  is  most  obvious  in  panel  1 
(Any  SACD  goal),  which  shows  that  86  percent  to  90  percent  of  control  teachers  reported  engaging  in  SACD 
activities  to  promote  any  SACD  goal  (and  more  than  50  percent  reported  engaging  in  SACD  activities  to 
promote  each  of  the  six  individual  SACD  goals).  More  than  half  of  the  control  teachers  reported  engaging  in 
such  activities  for  at  least  an  hour  a week  (52%  to  76%:  panel  2,  Any  SACD  goal)  and  36  percent  to  43 
percent  reported  engaging  in  any  activity  linked  to  a named  SACD  program  to  promote  any  SACD  goal 
(panel  3,  Any  SACD  goal). 


26  Intensity  was  defined  as  being  for  at  least  1 hour  per  week  because  this  was  about  the  amount  of  time  that  the  SACD 
programs  being  evaluated  considered  desirable  as  part  of  their  models. 
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Panel  1 : Engagement  in  any  activities  to  promote  SACD  goals 


SACD  activity 

Year  1 

(Spring  3rd  grade) 

Year  2 

(Spring  4th  grade) 

Year  3 

(Spring  5th  grade) 

T reat- 
ment 

Control  Impact 

p-value 

T reat- 
ment 

Control  Impact 

p-value 

T reat- 
ment 

Control  Impact 

p-value 

Teacher  sample  size 

441 

409 

425 

404 

425 

396 

Violence  prevention  and  peace  promotion  (percent) 

76.7* 

64.7 

12.0 

0.005 

76.1* 

64.3 

11.8 

0.002 

76.9* 

63.0 

13.9 

0.001 

Social  and  emotional  development  (percent) 

81.3* 

64.2 

17.1 

0.000 

84.4* 

60.4 

24.0 

0.000 

81.1* 

62.5 

18.6 

0.000 

Character  education  (percent) 

92.3* 

79.3 

13.0 

0.000 

92.3* 

75.7 

16.6 

0.000 

86.8* 

72.6 

14.2 

0.000 

Tolerance  and  diversity  (percent) 

75.7* 

62.1 

13.6 

0.001 

75.7* 

59.8 

15.9 

0.000 

73.8* 

57.4 

16.4 

0.000 

Risk  prevention  and  health  promotion  (percent) 

60.9 

62.2 

-1.3 

0.795 

63.5 

61.9 

1.6 

0.744 

63.7 

62.3 

1.4 

0.778 

Civic  responsibility  and  community  service  (percent) 

63.4 

61.7 

1.7 

0.687 

60.2 

58.4 

1.8 

0.669 

60.3 

59.1 

1.2 

0.795 

Any  SACD  goal  (percent) 

95.3* 

88.2 

7.1 

0.006 

95.4* 

90.3 

5.1 

0.007 

95.5* 

86.0 

9.5 

0.000 

Behavior  management  (percent) 

88.5 

85.6 

2.9 

0.285 

92.3* 

82.1 

10.2 

0.000 

85.0 

82.4 

2.6 

0.389 

Panel  2:  Engagement  in  any  activities  to  promote  SACD  goals  for  at  least  1 hour  per  week 


SACD  activity 

Year  1 

(Spring  3rd  grade) 

Year  2 

(Spring  4th  grade) 

Year  3 

(Spring  5th  grade) 

T reat- 
ment 

Control  Impact 

p-value 

T reat- 
ment 

Control  Impact 

p-value 

T reat- 
ment 

Control  Impact 

p-value 

Teacher  sample  size 

441 

409 

425 

404 

425 

396 

Violence  prevention  and  peace  promotion  (percent) 

39.4* 

21.9 

17.5 

0.000 

41.9* 

24.9 

17.0 

0.000 

38.0* 

24.2 

13.8 

0.002 

Social  and  emotional  development  (percent) 

46.9* 

23.4 

23.5 

0.000 

49.5* 

24.7 

24.8 

0.000 

42.6* 

25.9 

16.7 

0.000 

Character  education  (percent) 

60.9* 

31.3 

29.5 

0.000 

60.4* 

30.6 

29.8 

0.000 

50.2* 

30.5 

19.7 

0.000 

Tolerance  and  diversity  (percent) 

36.7* 

21.3 

15.4 

0.000 

36.0* 

23.1 

12.9 

0.001 

32.8* 

18.4 

14.4 

0.000 

Risk  prevention  and  health  promotion  (percent) 

25.9 

20.5 

5.4 

0.138 

28.0 

24.1 

3.9 

0.293 

24.5 

22.8 

1.8 

0.657 

Civic  responsibility  and  community  service  (percent) 

18.8 

14.7 

4.0 

0.179 

19.7 

16.1 

3.6 

0.252 

20. 6A 

14.6 

6.0 

0.087 

Any  SACD  goal  (percent) 

75.2* 

51.7 

23.5 

0.000 

83.7* 

72.6 

11.1 

0.003 

83.0* 

76.0 

7.0 

0.037 

Behavior  management  (percent) 

68.7 

66.2 

2.5 

0.666 

75.0* 

58.9 

16.1 

0.001 

68.6 

64.9 

3.7 

0.377 

See  notes  at  end  of  table. 
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Table  1 .20.  Impacts  on  teacher-reported  SACD  classroom  activities — Continued 

Panel  3:  Engagement  in  activities  to  promote  SACD  goals  linked  to  named  SACD  programs 


SACD  activity 

Year  1 

(Spring  3rd  grade) 

Year  2 

(Spring  4th  grade) 

Year  3 

(Spring  5th  grade) 

T reat- 
ment 

Control  Impact 

p-value 

T reat- 
ment 

Control  Impact 

p-value 

T reat- 
ment 

Control  Impact 

p-value 

Teacher  sample  size 

441 

409 

425 

404 

425 

396 

Violence  prevention  and  peace  promotion  (percent) 

42.0* 

17.0 

25.0 

0.000 

48.1* 

17.3 

30.8 

0.000 

48.0* 

17.4 

30.6 

0.000 

Social  and  emotional  development  (percent) 

48.9* 

8.3 

40.7 

0.000 

54.8* 

11.2 

43.6 

0.000 

48.8* 

13.0 

35.7 

0.000 

Character  education  (percent) 

54.4* 

15.5 

38.8 

0.000 

58.8* 

13.4 

45.4 

0.000 

49.4* 

13.3 

36.1 

0.000 

Tolerance  and  diversity  (percent) 

34.3* 

6.2 

28.2 

0.000 

37.6* 

7.2 

30.4 

0.000 

33.4* 

5.8 

27.7 

0.000 

Risk  prevention  and  health  promotion  (percent) 

27.7 

21.7 

6.0 

0.151 

31.3* 

21.7 

9.6 

0.013 

28.5* 

20.1 

8.4 

0.048 

Civic  responsibility  and  community  service  (percent) 

r 

t 

4.4 

0.009 

13.3* 

4.7 

8.6 

0.000 

13.5* 

7.2 

6.4 

0.021 

Any  SACD  goal  (percent) 

70.0* 

36.3 

33.6 

0.000 

72.4* 

42.6 

29.8 

0.000 

68.4* 

39.7 

28.7 

0.000 

Panel  4:  Engagement  in  activities  to  promote  SACD  goals  linked  to  named  SACD  programs  for  at  least  1 hour  per  week 


SACD  activity 

Year  1 

(Spring  3rd  grade) 

Year  2 

(Spring  4th  grade) 

Year  3 

(Spring  5th  grade) 

T reat- 
ment 

Control  Impact 

p-value 

T reat- 
ment 

Control  Impact 

p-value 

T reat- 
ment 

Control  Impact 

p-value 

Teacher  sample  size 

441 

409 

425 

404 

425 

396 

Violence  prevention  and  peace  promotion  (percent) 

29.4* 

8.5 

20.9 

0.000 

31.5* 

11.2 

20.3 

0.000 

29.2* 

10.1 

19.1 

0.000 

Social  and  emotional  development 

34.6* 

4.0 

30.7 

0.000 

36.9* 

6.3 

30.6 

0.000 

30.0* 

6.9 

23.1 

0.000 

Character  education  (percent) 

41.3* 

6.3 

35.0 

0.000 

43.3* 

6.7 

36.5 

0.000 

33.2* 

7.0 

26.2 

0.000 

Tolerance  and  diversity  (percent) 

22.8* 

1.4 

21.3 

0.000 

24.1* 

3.4 

20.7 

0.000 

22.0* 

2.7 

19.3 

0.000 

Risk  prevention  and  health  promotion  (percent) 

15.8 

11.2 

4.6 

0.132 

18.4 

13.9 

4.5 

0.197 

15.8* 

9.3 

6.5 

0.043 

Civic  responsibility  and  community  service  (percent) 

t* 

t 

3.5 

0.021 

7.6* 

1.5 

6.1 

0.010 

8.1* 

1.3 

6.8 

0.024 

Any  SACD  goal  (percent) 

52.7* 

18.6 

34.1 

0.000 

51.8* 

23.4 

28.5 

0.000 

46.0* 

19.6 

26.3 

0.000 

$ Reporting  standards  not  met.  Value  suppressed  to  protect  confidentiality. 

* Treatment  group  significantly  different  from  control  group  at  the  .05  level. 

A Treatment  group  significantly  different  from  control  group  at  the  .10  to  > .05  level. 

NOTE:  Impact  is  the  percentile  point  difference  between  treatment  and  control  groups.  Weights,  which  assign  equal  weight  to  each  school  within  each  of  the  programs  and  to  each 
program  across  programs,  were  used  in  producing  the  treatment,  control,  and  overall  means. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Use  of  Materials  and  Strategies 

The  SACD  programs  had  a statistically  significant  positive  effect  on  teacher-reported  use  of  specific  materials 
and  classroom  strategies  to  promote  social  and  character  development  (table  1.21).  Over  the  3 years,  87 
comparisons  were  made  between  treatment  and  control  teacher  use  of  materials  and  strategies,  with  4 
expected  to  be  significant  by  chance.  A significantly  greater  percentage  of  treatment  teachers  reported  using 
these  materials  and  strategies  to  support  SACD  goals  in  40  of  these  comparisons.  Treatment  teachers  were 
significantly  more  likely  than  control  teachers  to  report  use  of  four  of  the  five  specific  materials  asked  about 
in  at  least  2 of  the  3 years.  In  all  years,  treatment  teachers  were  significantly  more  likely  than  control  teachers 
to  report  using  teacher  guides  (by  22  to  24  percentage  points)  and  children’s  literature  (by  9 to  1 5 percentage 
points).  In  Years  2 and  3,  treatment  teachers  were  significantly  more  likely  to  report  using  student  materials 
and  instructional  aids  (by  10  to  12  percentage  points).  In  addition,  treatment  teachers  were  significantly  less 
likely  to  report  not  using  any  of  these  materials  (by  7 to  10  percentage  points)  in  all  years.  Control  teachers 
reported  rates  of  usage  of  materials  from  36  percent  for  instructional  aids  to  60  percent  for  teacher  guides, 
and  they  reported  higher  rates  for  not  using  any  of  the  materials  (13%  to  14%). 

Of  the  20  teaching  strategies  asked  about,  13  strategies  were  reported  as  being  used  by  a statistically 
significant  greater  percentage  of  treatment  teachers  (2  to  22  percentage  points  more)  for  at  least  one  of  the 
years,  and  no  strategy  was  reported  as  being  used  by  a significantly  greater  percentage  of  control  teachers. 
Treatment  teachers  reported  using,  on  average,  one  more  strategy  than  control  teachers,  and  this  was  a 
significant  difference. 
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Table  1 .21 . Impacts  on  use  of  SACD  classroom  materials  and  teaching  strategies 


Year  1 

(Spring  3rd  grade) 

Year  2 

(Spring  4th  grade) 

Year  3 

(Spring  5th  grade) 

SACD  material  and  teaching  strategy 

T reat- 
ment 

Control  Impact 

p-value 

Treat- 

ment 

Control  Impact 

p-value 

T reat- 
ment 

Control 

Impact 

p-value 

Teacher  sample  size 

441 

409 

425 

404 

425 

396 

Use  of  SACD  materials  (percent) 

Teacher  guides  (manuals,  curricula) 

82.4* 

58.1 

24.2 

0.000 

82.3* 

60.4 

21.8 

0.000 

79.9* 

58.0 

21.9 

0.000 

Student  materials  (workbooks  or  sheets) 

61.3 

57.0 

4.3 

0.289 

68.3* 

56.1 

12.2 

0.002 

66.2* 

55.2 

11.0 

0.002 

Instructional  aids  (games,  software,  videos) 

43.9 

39.1 

4.9 

0.294 

46.3* 

36.7 

9.6 

0.017 

47.6* 

35.7 

11.9 

0.001 

Giveaways  (bookmarks,  stickers) 

52.2 

54.2 

-1.9 

0.688 

52.9 

47.3 

5.6 

0.157 

49.9 

52.6 

-2.7 

0.485 

Children’s  literature 

60.7* 

46.2 

14.6 

0.002 

60.4* 

46.5 

13.9 

0.002 

53.4* 

44.6 

8.8 

0.013 

Other  types  of  materials 

8.2 

8.5 

-0.4 

0.854 

9.1 

12.2 

-3.1 

0.201 

9.0 

6.9 

2.1 

0.360 

Did  not  use  any  of  these  materials 

3.8* 

14.2 

-10.4 

0.001 

4.3* 

13.5 

-9.2 

0.000 

6.3* 

13.1 

-6.7 

0.003 

Use  of  teaching  strategies  (percent) 

Role-playing 

85.5* 

65.2 

20.3 

0.000 

86.6* 

65.1 

21.5 

0.000 

87.1* 

80.0 

7.1 

0.028 

Cooperative  learning 

96.3 

96.1 

0.2 

0.883 

98.0 

96.8 

1.3 

0.250 

99.8* 

97.7 

2.1 

0.022 

Peer  group  discussions 

92.9* 

87.7 

5.2 

0.011 

92. 5A 

88.2 

4.3 

0.060 

97. 5A 

95.1 

2.4 

0.096 

Direct  instruction  of  SACD 

91.8* 

78.8 

13.0 

0.000 

93.6* 

75.9 

17.6 

0.000 

97.1* 

87.9 

9.2 

0.000 

Skill  training 

63.0* 

43.2 

19.8 

0.000 

62.9* 

40.5 

22.4 

0.000 

82.1* 

75.5 

6.6 

0.037 

Incorporating  SACD  into  academic  curriculum 

86.5* 

73.6 

12.9 

0.000 

85.3* 

71.3 

14.0 

0.000 

92.9 

90.3 

2.5 

0.225 

Parent  training 

8.1 

8.9 

-0.8 

0.702 

8.6 

7.2 

1.4 

0.531 

30.8* 

21.5 

9.3 

0.016 

Parent/community  involvement 

22.9 

27.0 

-4.1 

0.237 

26. 5A 

19.8 

6.7 

0.087 

48.3* 

39.7 

8.6 

0.022 

Mentoring 

44.8 

44.5 

0.3 

0.909 

43.8 

43.1 

0.7 

0.821 

68.1* 

58.3 

9.8 

0.016 

Good  behavior  notes  sent  home  daily  or  weekly 

75.8 

72.5 

3.3 

0.409 

78.2 

71.9 

6.3 

0.108 

91.7 

89.5 

2.1 

0.270 

Presenting  role  models 

76.4 

73.3 

3.1 

0.335 

75.7* 

68.4 

7.2 

0.026 

85.3 

81.6 

3.7 

0.175 

See  notes  at  end  of  table. 
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Table  1.21.  Impacts  on  use  of  SACD  classroom  materials  and  teaching  strategies — Continued 


Year  1 

(Spring  3rd  grade) 

Year  2 

(Spring  4th  grade) 

Year  3 

(Spring  5th  grade) 

SACD  material  and  teaching  strategy 

T reat- 
ment 

Control  Impact 

p-value 

T reat- 
ment 

Control  Impact  p-value 

T reat- 
ment 

Control  Impact 

p-value 

Use  of  teaching  strategies — Continued  (percent) 
Targeted  story  reading  or  writing  on  social 
and  character  development  themes 

88.2* 

75.9 

12.3 

0.000 

88.2* 

78.0 

10.2 

0.000 

93.6 

90.6 

3.0 

0.125 

Peer  mediation 

51.2 

45.8 

5.4 

0.205 

54.4 

48.5 

6.0 

0.190 

68.8 

65.0 

3.8 

0.337 

Honor  roll  for  positive  behavior 

55.1 

57.9 

-2.9 

0.569 

59.8 

57.2 

2.6 

0.585 

73.6 

68.3 

5.3 

0.235 

Pledges  or  recitations  on  social  and 
character  development  themes 

41.3 

42.8 

-1.4 

0.802 

43.4 

44.6 

-1.2 

0.849 

57.3 

59.1 

-1.8 

0.721 

Guided  visualization 

60.9* 

45.7 

15.2 

0.000 

60.5* 

47.3 

13.2 

0.003 

70.8* 

59.0 

11.9 

0.001 

Student-led/student-assisted  instruction 

56. 0A 

49.8 

6.2 

0.094 

57.3* 

49.0 

8.3 

0.028 

75.9 

70.3 

5.6 

0.113 

Journaling 

77.7 

72.3 

5.5 

0.138 

76.8 

71.4 

5.4 

0.149 

87.1 A 

82.1 

5.0 

0.093 

Time  out  for  negative  behavior 

86.2 

85.3 

0.8 

0.803 

85.5 

83.0 

2.4 

0.349 

94.1* 

89.0 

5.1 

0.040 

Daily  or  weekly  rewards  for  positive 
behavior 

91. 0A 

86.8 

4.1 

0.098 

92.1 

89.7 

2.4 

0.307 

98. 2A 

96.0 

2.2 

0.092 

Any  strategy 

100.0 

99.7 

0.3 

t 

100.0 

100.0 

0.0 

t 

99.8 

99.4 

0.4 

0.429 

Number  of  strategies  (mean) 

13.4* 

12.3 

1.1 

0.000 

13.6* 

12.1 

1.5 

0.000 

15.7* 

14.6 

1.1 

0.000 

t Not  applicable. 

* Treatment  group  significantly  different  from  control  group  at  the  .05  level. 

A Treatment  group  significantly  different  from  control  group  at  the  .10  to  > .05  level. 

NOTE:  Weights,  which  assign  equal  weight  to  each  school  within  each  of  the  programs  and  to  each  program  across  programs,  were  used  in  producing  the  treatment,  control,  and 
overall  means. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Use  of  Schoolwide  Strategies 

The  SACD  programs  did  not  have  an  impact  on  the  reported  use  of  schoolwide  strategies  for  supporting 
social  and  character  development.  Teachers  were  asked  to  report  on  the  use  of  six  schoolwide  strategies  to 
promote  social  and  character  development,  including  morning  announcements  or  videos,  assemblies, 
newspapers  or  bulletins,  special  school  days,  special  events,  and  other  activities.  Over  the  3 years,  18 
comparisons  were  made,  with  1 expected  to  be  significant  by  chance.  The  SACD  programs  did  not  have  a 
statistically  significant  effect  on  the  use  of  any  of  these  schoolwide  strategies  (these  data  are  not  shown  in  a 
table). 

Participation  in  Professional  Development 

The  SACD  programs  had  a statistically  significant  positive  effect  on  teachers’  reports  of  participation  in 
professional  development  for  activities  related  to  social  and  character  development.  Over  the  3 years,  27 
comparisons  were  made,  with  1 expected  to  be  significant  by  chance.  A significandy  greater  percentage  of 
treatment  teachers  reported  such  professional  development  in  13  of  the  comparisons  (table  1.22).  The 
percentage  difference  in  treatment  and  control  teacher  reports  on  receiving  SACD  training  in  the  previous  12 
months  ranged  from  32  percentage  points  in  Year  1 to  9 percentage  points  in  Year  3.  Mirroring  the  decline  in 
effect  over  time,  only  in  Year  1 was  there  a significant  impact  on  the  number  of  hours  of  professional 
development  received  (9  hours  versus  4 hours).  The  significance  of  the  impacts  on  percentages  of  teachers 
receiving  training  to  help  meet  each  of  the  six  SACD  goals  varied  by  year:  impacts  were  significant  for  four 
goals  in  Year  1,  one  goal  in  Year  2,  and  three  goals  in  Year  3,  with  the  differences  ranging  from  7 to  34 
percentage  points.  In  Year  1,  there  was  a significant  difference  in  reports  on  behavior  management  training 
(11  percentage  points). 
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Table  1.22.  Impacts  on  teacher-reported  SACD  professional  development 


Year  1 

(Spring  3rd  grade) 

Year  2 

(Spring  4th  grade) 

Year  3 

(Spring  5th  grade) 

SACD  professional  development 

T reat- 
ment 

Control  Impact 

p-value 

T reat- 
ment 

Control  Impact 

p-value 

T reat- 
ment 

Control  Impact 

p-value 

Teacher  sample  size 

441 

409 

425 

404 

425 

396 

SACD  training  in  past  12  months  (percent) 

85.6* 

53.8 

31.8 

0.000 

64.1* 

52.4 

11.7 

0.003 

56.6* 

47.3 

9.3 

0.027 

Hours  of  SACD  training  (mean) 

9.1* 

4.3 

4.9 

0.000 

5.3 

4.8 

0.6 

0.580 

4.5 

3.7 

0.8 

0.196 

Training  by  goal  (percent) 

Violence  prevention  and  peace  promotion 

26.9* 

17.3 

9.6 

0.009 

18.2 

14.3 

3.9 

0.241 

17.8* 

10.0 

7.8 

0.012 

Social  and  emotional  development 

36.3* 

19.4 

17.0 

0.000 

22.6 

18.0 

4.6 

0.209 

18.2* 

11.2 

7.0 

0.044 

Character  education 

60.9* 

26.7 

34.2 

0.000 

36.2* 

19.3 

16.9 

0.000 

28.0* 

14.8 

13.2 

0.001 

Tolerance  and  diversity 

21.0* 

12.3 

8.7 

0.017 

16.8 

19.0 

-2.2 

0.486 

16.8 

15.5 

1.3 

0.741 

Risk  prevention  and  health  promotion 

13.3 

14.3 

-1.0 

0.738 

13.6 

14.8 

-1.2 

0.707 

14.4 

12.0 

2.4 

0.404 

Civic  responsibility  and  community  service 

9.8 

7.0 

2.7 

0.221 

7.5 

4.8 

2.7 

0.130 

3.9 

4.8 

-0.9 

0.562 

Behavior  management 

36.4* 

25.0 

11.4 

0.023 

26.7 

25.6 

1.1 

0.784 

23.5 

21.4 

2.1 

0.613 

* Treatment  group  significantly  different  from  control  group  at  the  .05  level. 

NOTE:  Weights,  which  assign  equal  weight  to  each  school  within  each  of  the  programs  and  to  each  program  across  programs,  were  used  in  producing  the  treatment,  control,  and 
overall  means. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Attitudes  and  Practices 

The  SACD  programs  had  a statistically  significant  positive  effect  on  teachers’  reports  on  the  use  of  six 
practices27  considered  conducive  to  the  social  and  character  development  of  students  but  not  on  teacher 
attitudes  toward  social  and  character  development.  Twenty-seven  comparisons  were  made  over  the  3 years, 
with  1 expected  to  be  found  significant  by  chance.  A significantly  greater  percentage  of  treatment  teachers 
reported  that  these  practices  were  being  used  in  4 of  these  comparisons  (these  data  are  not  shown  in  a table). 
In  all  3 years,  a significandy  larger  percentage  of  treatment  teachers  reported  students  having  a voice  in  school 
governance  (9%  to  10%  versus  4%  to  5%  of  control  teachers).  In  Year  3,  treatment  teachers  reported 
significantly  greater  use  of  discipline  for  promoting  development  (57%  versus  47%  of  control  teachers). 

There  were  no  statistically  significant  estimated  impacts  on  teachers’  enthusiasm  for  SACD  efforts  in  their 
schools  in  any  of  the  years  (these  data  are  not  shown  in  a table).  For  the  3 years,  between  67  percent  and  71 
percent  of  treatment  and  control  teachers  reported  that  they  were  enthusiastic  about  their  schools’  efforts  to 
promote  social  and  character  development  and  another  27  percent  to  31  percent  reported  being  cooperative 
(but  not  enthusiastic).  Fewer  than  2 percent  reported  an  open  dislike  of  the  SACD  efforts  in  their  schools. 

Domain  Results 

As  discussed  in  the  Analysis  section,  a set  of  heuristics  was  applied  to  determine  whether  the  six  outcome 
domains  were  statistically  significant  after  adjustments  were  made  for  the  multiple  tests  performed  under  each 
domain.  These  heuristics  were  applied  to  both  the  combined-program  data  and  to  each  individual  program’s 
data.  Table  1.23  shows  the  statistically  significant  impacts  on  the  use  of  SACD  activities  by  domain  for  all 
seven  programs  combined  and  for  each  program.  In  table  1.23,  a plus  sign  stands  for  a significant  positive 
impact  on  the  domain,  and  numeral  superscripts  show  which  heuristics  identified  the  domain  as  significant. 
For  all  seven  programs  combined,  a significant  positive  impact  was  found  for  each  of  the  3 years  on  the 
reports  of  use  of  any  SACD  activities,  the  use  of  SACD  activities  linked  to  named  SACD  programs,  the  use 
of  classroom  materials  and  teaching  strategies,  and  professional  development. 

Table  1.23  also  shows  the  impacts  on  domain  by  individual  program.  In  all  cases,  when  a program’s  impact 
was  significant  it  was  also  positive.  Five  of  the  programs  had  at  least  2 years  of  significant  impacts  on 
teachers’  reported  use  of  activities,  as  did  six  programs  for  teachers’  use  of  activities  linked  to  named  SACD 
programs.  Two  programs  had  at  least  2 years  of  significant  impacts  on  reported  use  of  classroom  materials 
and  strategies.  No  programs  showed  impacts  on  reported  use  of  schoolwide  strategies  or  on  attitudes  and 
practices  in  at  least  2 years.  One  program  showed  2 years  of  impacts  on  reported  professional  development. 

Results  From  Principal  Interviews 

The  principal  interviews  in  2005, 28  2006,  and  2007  echoed  the  findings  from  the  teacher  survey  that  there  was 
a high  usage  of  SACD  activities  in  both  the  treatment  and  control  schools  (these  data  are  not  shown  in  a 
table).  Over  the  years,  88  percent  to  100  percent  of  treatment  principals  and  71  percent  to  98  percent  of 
control  principals  reported  that  their  schools  used  SACD  activities  to  promote  each  of  the  six  SACD  goals. 
Classroom  teachers  were  reported  as  having  a major  role  in  delivering  SACD  activities  by  91  percent  to  98 
percent  of  treatment  school  principals  and  64  percent  to  90  percent  of  control  school  principals.  SACD 
activities  were  being  provided  universally  to  all  students  rather  than  to  targeted  populations,  as  reported  by  98 


27  These  included  teacher  modeling  of  positive  character  and  behavior  with  students,  similar  modeling  with  other  staff, 
involvement  of  students  in  discussions,  students  having  a voice  in  governance,  school  encouragement  of  parent 
involvement,  and  discipline  practices  that  included  promoting  development. 

28  Only  the  results  from  the  closed-ended  items  are  reported  for  Year  1 because  of  problems  with  inter-rater  reliability 
regarding  the  open-ended  items.  Results  from  both  the  closed-  and  open-ended  items  are  reported  for  Years  2 and  3. 
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Table  1.23.  Significant  impacts  on  use  of  SACD  activity  domains,  overall  and  by  program 


SACD  activity  domain 

Named 

Classroom 

SACD 

materials 

Attitudes 

Any  SACD 

program 

and  Schoolwide 

Professional 

and 

Program 

activities 

activities 

strategies  strategies 

development 

practices 

Overall 

Year  1 

1,2,3 

1,2,3 

+2,3 

+1,2,3 

Year  2 

1,2,3 

1,2,3 

1,2,3 

+2,3 

Year  3 

1,2,3 

+ 1,2,3 

+ 1,2,3 

+2,3 

ABC 

Year  1 

2 

+ 

1 

+ 

Year  2 

Year  3 

+3 

CSP 

Year  1 

2 

+ 

+1'3 

+3 

Year  2 

+3 

Year  3 

+3 

LBW 

Year  1 

+2'3 

+3 

+3 

+3 

Year  2 

+3 

+3 

+2 

Year  3 

PA 

Year  1 

+2,3 

+1'3 

+3 

+3 

Year  2 

+1'3 

Year  3 

+3 

+3 

PATHS 

Year  1 

+2,3 

+1'3 

+1’3 

Year  2 

+3 

+1'3 

Year  3 

+1'3 

4Rs 

Year  1 

+1,2,3 

+ 1,2,3 

+3 

+1'3 

Year  2 

+1'3 

+1'3 

Year  3 

+1’3 

+ 1,2,3 

+3 

+3 

See  notes  at  end  of  table. 
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Table  1.23. 

Significant  impacts  on  use  of  SACD  activity  domains,  overall  and  by  program — 
Continued 

SACD  activity  domain 

Named 

Classroom 

SACD 

materials 

Attitudes 

Any  SACD 

program 

and 

Schoolwide 

Professional 

and 

Program 

activities 

activities 

strategies 

strategies 

development 

practices 

SS 

Year  1 

+3 

+1’3 

+3 

Year  2 

+1’3 

+1’3 

+3 

Year  3 

+3 

+1’3 

+3 

1 Based  on  univariate  statistical  tests,  at  least  half  of  the  impacts  were  positive  and  statistically  significant  and  no  impact  was 
negative  and  statistically  significant. 

2 The  omnibus  impact  for  all  the  outcomes  measured  together  was  positive  and  statistically  significant  on  the  basis  of  a multivariate 
statistical  test. 

3 At  least  one  outcome  remained  positive  and  statistically  significant  and  no  outcome  was  negative  and  statistically  significant  after 
applying  the  Benjamini-Hochberg  (1 995)  procedure  to  adjust  significance  levels  downward  to  account  for  the  multiple  testing  of 
impacts. 

NOTE:  Abbreviations  are 

ABC:  Academic  and  Behavioral  Competencies  Program 
CSP:  Competence  Support  Program 
LBW:  Love  In  a Big  World 
PA:  Positive  Action 

PATHS:  Promoting  Alternative  Thinking  Strategies 

4Rs:  The  4Rs  Program  (Reading,  Writing,  Respecting,  and  Resolution) 

SS:  Second  Step 

+:  Statistically  significant  beneficial  impact  on  domain 
Blank  cell:  Finding  of  no  impact 

Significance  is  based  on  p < .05.  No  detrimental  impact  was  found  statistically  significant  at  or  below  the  .05  level.  Description  of 
SACD  activity  domains  and  the  heuristics  used  to  determine  the  statistically  significant  beneficial  impact  on  the  domain. 

SACD  activities:  based  on  16  teacher-reported  measures  on  the  use  of  SACD  activities  in  the  classroom. 

SACD  activities  linked  to  named  programs:  based  on  14  teacher-reported  measures  on  the  use  of  SACD  activities  associated 
with  a named  program  in  the  classroom. 

Classroom  materials  and  strategies:  based  on  29  teacher-reported  measures,  7 concerning  materials  used  in  the  classroom 
and  22  concerning  classroom  strategies. 

Schoolwide  strategies:  based  on  six  teacher-reported  measures  concerning  strategies  to  promote  SACD  schoolwide. 

Professional  development:  based  on  nine  teacher-reported  measures  concerning  their  participation  in  SACD-related  training 
Attitudes  and  practices:  based  on  nine  teacher-reported  measures,  three  concerning  teacher  attitudes  toward  SACD  efforts  in  the 
school  and  six  concerning  school  practices  conducive  to  the  social  and  character  development  of  students. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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percent  to  100  percent  of  treatment  school  principals  and  91  percent  to  94  percent  of  control  school 
principals.  Unlike  the  findings  from  the  teacher  surveys,  the  principal  surveys  did  not  show  a significandy 
greater  use  of  SACD  activities  among  the  treatment  schools,  although  this  might  be  due  to  the  inability  to 
perform  significance  tests  in  seven  cases  as  discussed  below.  In  regard  to  whether  principals  of  treatment 
schools  reported  greater  use  of  activities  to  promote  each  of  the  six  SACD  goals  than  principals  of  control 
schools,  18  comparisons  were  made  over  the  3 years,  with  1 expected  to  be  significant  by  chance.  In  two 
cases  (both  in  Year  2),  a significandy  greater  percentage  of  treatment  principals  reported  the  use  of  SACD 
activities  to  support  SACD  goals:  (1)  tolerance  and  diversity  (98%  of  treatment  school  principals  versus  83% 
of  control  school  principals,  with  a Rvalue  of  0.036)  and  (2)  risk  prevention  and  health  promotion  (88% 
versus  71%,  with  a ^-value  of  0.044).  These  results  do  not  provide  evidence  in  support  of  the  SACD 
programs’  impact  on  schools’  use  of  SACD  activities  because  they  are  small  in  number  and  are  not  replicated 
over  multiple  years.  However,  for  five  of  the  comparisons  significance  testing  could  not  be  done  because  100 
percent  of  the  treatment  principals  reported  that  their  schools  had  activities  promoting  a specific  SACD  goal. 

In  regard  to  staff  development,  27  comparisons  were  made  over  the  3 years,  with  1 expected  to  be  found 
significant  by  chance.  In  two  cases  (both  in  Year  2),  there  was  a significant  difference  favoring  the  treatment 
schools.  First,  the  mean  percentage  of  teachers  per  school  reported  by  their  principals  to  have  received 
training  in  social  and  character  development  in  the  past  year  was  significantly  higher  for  treatment  schools 
versus  control  schools  (55%  versus  31%,  with  a Rvalue  of  0.003).  Second,  the  percentage  of  principals 
reporting  that  only  0 percent  to  25  percent  of  their  faculty  had  received  such  training  treatment  was 
significantly  smaller  among  treatment  school  principals  than  control  school  principals  (33%  versus  67%,  with 
a 7>-value  of  0.002).  These  results  do  not  provide  evidence  in  support  of  the  SACD  programs’  impact  on 
teacher  training  because  they  are  small  in  number  and  are  not  replicated  over  multiple  years.  However,  for 
two  of  the  comparisons  significance  testing  could  not  be  done  because  100  percent  of  the  treatment 
principals  gave  the  same  report. 

Findings  Regarding  Use  of  SACD  Activities 

The  analysis  of  use  of  SACD  activities  had  two  main  findings:  (1)  the  SACD  programs  increased  the  reported 
use  of  and  training  in  activities  to  promote  students’  social  and  character  development  in  the  classroom 
compared  to  what  reportedly  occurred  in  the  control  schools,  and  (2)  the  reports  from  the  control  schools 
identified  them  as  a standard  practice  control  in  that  their  staffs  reported  the  use  of  and  training  in  similar 
activities  as  the  treatment  staff. 

Evidence  for  the  first  finding  came  primarily  from  the  teacher  reports.  Treatment  teachers  reported 
significantly  greater  implementation  of  activities  than  control  teachers  for  127  of  the  249  outcomes  measured 
and  for  four  of  the  six  SACD  activity  domains  during  the  3 years.  Evidence  for  the  second  finding  came  from 
both  the  principal  and  teacher  reports.  The  principals  reported  similar  levels  of  SACD  activities  and  teacher 
training  among  treatment  and  control  schools  (though  statistical  difficulties  in  testing  all  the  comparisons  may 
have  contributed  to  this  finding).  The  treatment  teachers’  reports  were  statistically  significantly  greater  than 
those  of  the  control  teachers  but  the  practical  differences  varied.  For  example,  the  significantly  greater 
percentage  of  treatment  teachers  reporting  use  of  activities  to  promote  any  SACD  goal  was  5 to  10 
percentage  points  greater  than  the  percentage  of  control  teachers  (86%  to  90%  of  whom  reported  engaging  in 
such  activities),  while  the  significant  difference  for  the  reported  use  of  activities  linked  to  named  programs  to 
promote  any  SACD  goal  ranged  between  29  and  34  percentage  points  (with  36%  to  42%  of  control  teachers 
reporting  engaging  in  these  activities). 
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Year-by-Year  Impacts  on  Students  and  Perceptions  of  School  Climate 

The  second  and  third  of  the  primary  research  questions  for  the  SACD  evaluation  were  as  follows: 

What  is  the  average  effect  of  the  seven  universal,  school-based,  social  and  character 
development  programs  on  students’  social  and  emotional  competence,  behavior, 
academics,  and  on  perceptions  of  school  climate? 

What  is  the  average  effect  of  each  specific  SACD  program  on  students’  social  and 
emotional  competence,  behavior,  academics,  and  on  perceptions  of  school  climate? 

One  method  to  answer  these  questions  was  to  examine  the  year-by-year  impacts  of  the  SACD  programs  on 
these  outcomes  over  the  3 years  as  the  students  progressed  from  third  through  fifth  grades.  The  examination 
of  year-by-year  impacts  entailed  three  sets  of  analyses  resulting  in  three  sets  of  impacts.  The  first  set  of 
analyses  compared  the  outcomes  of  treatment  and  control  students  from  the  fall  of  third  grade  to  the  spring 
of  third  grade.  The  second  set  compared  the  outcomes  from  the  fall  of  third  grade  to  the  spring  of  fourth 
grade,  and  the  third  set  compared  the  fall  of  third  grade  to  the  spring  of  fifth  grade.  Within  each  set  of  year- 
by-year  analyses,  an  analysis  of  all  the  programs  together  provided  impact  results  for  the  set  of  seven 
coherent,  universal,  school-based  programs,  and  individual  program  analysis  provided  results  specific  to  each 
program.  In  answering  the  first  question  above,  the  analysis  of  the  combined  seven  programs  was  able  to 
detect  smaller  statistically  significant  impacts  because  of  its  larger  sample  size  and  associated  greater  power 
than  the  analysis  of  individual  SACD  programs,  which  were  based  on  the  smaller  samples  available  to  answer 
the  second  question  above. 

Analysis 

The  random  assignment  design  ensured  that  the  main  difference  between  the  treatment  and  control  group 
schools  at  the  point  of  random  assignment  was  exposure  to  the  mix  of  classroom-based  and  other  activities 
that  comprised  the  SACD  program  being  tested  in  each  site.  Thus,  unbiased  estimates  of  the  average  impacts 
of  the  SACD  programs  (relative  to  the  social  and  character  development  activities  offered  in  the  control 
schools)  could  be  computed  as  the  difference  in  the  average  outcomes  of  students  and  teachers  in  the 
treatment  and  control  schools.  Because  of  the  repeated  cross-sectional  design,  this  approach  yielded  unbiased 
estimates  of  the  combined  effects  of  the  SACD  programs  on  student  outcomes  and  on  potential  treatment- 
induced  school  entry  and  exit  effects  that  could  have  influenced  the  composition  of  students  and  teachers  in 
the  schools  at  the  follow-up  points. 

Statistical  precision  was  a concern  for  the  SACD  multiprogram  evaluation  because  of  design  effects  due  to 
the  clustering  of  students  within  schools  (students  shared  the  same  teachers  and  school  environment)  and  the 
relatively  small  numbers  of  schools  available  for  program-level  analyses.  Therefore,  regression  procedures 
were  used  rather  than  simple  differences-in-means  procedures  to  estimate  impacts  to  improve  the  statistical 
precision  of  the  estimates  and  to  adjust  for  differences  between  the  treatment  and  control  groups’  observable 
characteristics  due  to  random  selection,  study  nonconsent,  and  interview  nonresponse.29 

Two  analyses  are  discussed  in  this  section.  The  first  is  the  overall  analysis  combining  results  from  all 
programs.  The  overall  analysis  was  done  to  determine  if  providing  schools  with  a SACD  program  (as 
reflected  in  the  average  results  from  seven  different  programs)  improved  student  outcomes.  The  seven 


29  The  use  of  regression  models  raised  the  issue  of  the  distribution  of  the  outcome  variables.  Calculated p-values  rely  on 
the  assumption  of  normality,  and  a violation  of  this  condition  may  cause  biases  in p-values,  thus  leading  to  invalid 
hypothesis  testing.  However,  if  the  sample  size  is  sufficiently  large,  the  large  sample  approximation  for  the  distribution 
of  the  estimates  can  be  used  and  the  normality  assumption  can  be  relaxed  as  it  was  for  the  SACD  evaluation. 
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programs  sought  to  change  a common  set  of  student  behaviors  by  providing  school-based  instruction 
(drawing  upon  a common  set  of  methods  and  approaches)  through  the  teacher  to  all  students.  The  focus  of 
the  programs  differed  in  terms  of  specific  instructional  approaches  and  student  outcomes  so  they  did  not 
represent  a unified,  single  approach  to  improving  student  social  and  character  development.  Therefore  the 
combined-program  analysis  was  not  an  evaluation  of  a general  SACD  approach  (though  the  results  could 
contribute  to  such  an  evaluation)  nor  does  it  have  any  direct  application  to  similar  SACD  programs  not 
included  in  this  study.  Second,  individual  analysis  of  each  program  was  done  to  evaluate  them  separately. 
Besides  providing  an  evaluation  of  each  program,  these  analyses  were  used  to  examine  the  possibility  that 
results  for  a subset  of  the  programs  led  to  the  results  from  the  combined-program  analysis. 

The  Combined-Program  Model  and  the  Program-Level  Models 

For  the  combined-program  analysis,  pooled  impact  estimates  from  all  the  programs  were  obtained  from  each 
year  of  the  study  to  examine  the  extent  to  which,  taken  together,  the  seven  SACD  programs,  on  average, 
changed  student  and  school  outcomes  relative  to  what  they  would  have  been  otherwise.  A hierarchical  linear 
model  (HLM)  was  used  to  estimate  regression-adjusted  impacts  (Bryk  and  Raudenbush  1992).  The  basic 
model  consisted  of  two  levels  that  were  indexed  by  students  or  teachers  (?)  and  schools  (j),  and  where  fixed 
site  effects  were  indexed  by  d\ 

(1)  Level  1 : Students  / Teachers : Yjsd  = a0sd  + Xisdfi  + eisd 

Level  2 : Schools : a0sd  = y0  + yxTsd  + 9d  + ZJ  + usd . 

In  this  model,  Ylui  is  an  outcome  measure  (from  spring  of  Year  1,  Year  2,  or  Year  3)  for  a student  or  teacher; 
(Xosd  is  a school-level  random  intercept;  Xjs,i  are  student-  or  teacher-level  initial  covariates30;  Tsd  is  a binary 
variable  equal  to  1 for  treatment  group  schools  and  0 for  control  schools;  0d  are  program-specific  fixed  effects; 
Zsd  are  school-level  initial  covariates  (and  teacher-level  initial  covariates  for  student  outcomes)31;  fi,  Jo,  Yu  and 
5 are  fixed  parameter  vectors  to  be  estimated;  emi  are  assumed  to  be  independent  and  identically  distributed 
N(0,  <72e)  student-  or  teacher-level  random  error  terms;  and  usd  are  independent  and  identically  distributed 
N(0,  O2,,)  school-specific  error  terms  that  capture  the  correlation  between  the  outcomes  of  students  (teachers) 
in  the  same  schools  and  are  assumed  to  be  distributed  independently  of  the  Level  1 error  terms. 

Inserting  the  Level  2 equation  into  the  Level  1 equation  creates  the  following  single  level  model: 

(2)  Yisd=y o + yxTsd  + 9d  + Zsd5  + XisdjL  + [usd  + eisd  ] . 


In  this  formulation,  the  estimate  of  the  parameter,  yi,  is  the  regression-adjusted,  combined-program  impact 
estimate.  The  standard  error  of  this  estimate  accounts  for  design  effects  due  to  the  clustering  of  students  in 
schools,  as  well  as  precision  gains  from  the  inclusion  of  initial  covariates  that  explain  some  of  the  variation  in 


30  The  X vector  refers  to  initial  covariates  for  students  when  student  outcomes  are  used  and  to  initial  covariates  for 
teachers  when  teacher-level  outcomes  are  used.  For  new  entrants,  most  covariates  were  considered  time-invariate  and  so 
could  be  used  as  initial  values  when  collected  later  in  the  study.  Imputation  was  used  to  estimate  the  values  for  time- 
variate  covariates  (such  as  initial  scores  on  outcomes)  using  information  from  similar  students  who  were  in  the  sample  in 
Year  1.  In  addition,  for  students  in  the  sample  during  Year  1,  some  of  the  covariates,  such  as  the  student  outcome 
scores,  could  have  changed  between  the  start  of  program  implementation  and  data  collection  in  fall  2004.  Sensitivity 
analyses  reported  in  appendix  B show  that  the  point  estimates  of  the  impacts  are  similar  when  the  covariates  are 
included  or  excluded  in  the  model. 

31  Conceptually,  the  Z vector  referred  to  both  school-level  and  teacher-level  covariates.  However,  the  final  model  did  not 
use  school-level  covariates  so  Z actually  includes  only  teacher-level  initial  covariates  for  the  analysis  of  student  outcomes. 
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outcomes  between  and  within  schools.  T tests  were  used  to  gauge  the  statistical  significance  of  the  impact 
estimates. 

This  model  produces  impact  estimates  that  are  internally  valid  but  not  necessarily  externally  valid  to  a broader 
population  of  sites.  This  stems  from  treating  site  effects  as  fixed  rather  than  random  (between-site  variance 
terms  are  not  accounted  for  in  the  variance  calculations),  which  was  done  because  sites  were  selected 
intentionally  for  the  study.  In  addition,  because  all  classrooms  within  the  study  schools  were  included  in  the 
evaluation  (there  was  no  sampling  of  classrooms  and  the  model  does  not  include  classroom  error  terms),  the 
estimated  student-level  impact  findings  generalize  to  the  classrooms  in  the  study  schools  at  the  time  of  the 
evaluation. 

For  the  evaluation  of  each  program,  impact  estimates  were  generated  by  estimating  equation  (2)  separately  for 
each  program.  Each  model  included  program-specific  covariates  and  random  school  effects  but  excluded  the 
program  fixed  effects  (Od).  Sample  weights  were  used  in  all  analyses.  F tests  were  used  to  test  for  differences 
in  estimated  impacts  across  programs  and  / tests  were  used  to  determine  the  statistical  significance  of  the 
program-specific  impact  estimates. 

Covariates 

The  X and  Z covariates  in  equation  (2)  were  constructed  using  the  fall  2004  surveys.  The  covariates  were 
selected  based  on  two  main  criteria:  (1)  they  should  adjust  for  statistically  significant  treatment  and  control 
differences  at  initial  data  collection;  and  (2)  they  should  have  predictive  power  across  a broad  range  of 
outcomes.  Using  stepwise  regression  procedures,  a separate  set  of  covariates  were  selected  for  the  outcomes 
from  each  of  the  four  data  sources  for  the  combined-program  analysis  (see  appendix  B for  details).  Table  1.24 
lists  the  covariates  used  with  the  outcomes  from  each  report  (child,  primary  caregiver,  teacher  on  student,  and 
teacher  on  classroom  and  school).  These  covariates  were  used  for  the  analysis  of  the  Year  1,  Year  2,  and  Year 
3 data.  As  a sensitivity  check,  the  same  covariate  selection  process  was  used  with  the  Year  2 data  and  the 
model  re-estimated  using  the  alternative  covariates.  The  use  of  the  alternative  covariates  did  not  lead  to 
different  patterns  of  results,  although  the  statistical  significance  of  some  of  the  program-level  impact 
estimates  changed.  For  all  3 years,  missing  covariates  were  imputed  using  mean  values  for  nonmissing  cases, 
by  school,  gender,  and  race/ ethnicity. 

For  the  individual  evaluations  of  each  program,  covariates  were  identified  using  the  same  procedure  as  above 
but  were  allowed  to  vary  between  programs.  This  led  to  28  different  sets  of  covariates  (seven  programs  by 
four  reports  providing  outcomes).  The  covariates  used  in  the  models  for  the  specific  programs  are  detailed  in 
the  program  chapters. 
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Table  1.24.  Covariates  used  with  outcomes  from  each  report  for  combined-program  analysis 


Potential  covariate 

CR 

outcome 

PCR 

outcome 

TRS 

outcome 

TRCS 

outcome 

Total  number 

26 

35 

28 

8 

Child-reported 

Female 

Y 

Y 

Y 

Hispanic  (non-White) 

Y 

Y 

Y 

Black  (non-Hispanic) 

Y 

Y 

Y 

Other  ethnicity 

Y 

Y 

Y 

Age  in  years 

Y 

Y 

Y 

Scales 

Afraid  at  School 

Altruistic  Behavior 

Y 

Y 

Empathy 

Y 

Engagement  with  Learning 

Y 

Y 

Y 

Negative  School  Orientation 

Y 

Y 

Normative  Beliefs  About  Aggression 

Y 

Sense  of  School  as  a Community 

Y 

Y 

Problem  Behavior 

Y 

Self-Efficacy  for  Peer  Interactions 

Y 

Y 

Victimization  at  School 

Y 

Primary  caregiver-reported 

Age  in  years 

Completed  high  school  or  equivalent 

Y 

Y 

Y 

Some  college 

Y 

Y 

Y 

Bachelor’s  or  higher  degree 

Y 

Y 

Y 

Highest  level  of  education  in  household 

Completed  high  school  or  equivalent 

Y 

Y 

Y 

Some  college 

Y 

Y 

Y 

Bachelor’s  or  higher  degree 

Y 

Y 

Y 

Mother  present  in  home  life 

Y 

Y 

Mother  and  father  present 

Y 

Y 

Y 

Respondent  someone  other  than  mother  or  father 

Y 

Y 

Y 

Number  of  people  in  household 

Y 

Y 

Y 

Household  income:  $20,000  to  $40,000 

Y 

Y 

Household  income:  $40,000  to  $60,000 

Y 

Y 

Household  income:  More  than  $60,000 

Y 

Y 

Income-to-poverty-threshold  ratio:  Below  135  percent 

Y 

Y 

Income-to-poverty-threshold  ratio:  135  to  185  percent 

Y 

Y 

Full-time  employment 

Y 

Part-time  employment 

Y 

See  notes  at  end  of  table. 
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Table  1.24.  Covariates  used  with  outcomes  from  each  report  for  combined-program 
analysis — Continued 


Potential  covariate 

CR 

outcome 

PCR 

outcome 

TRS 

outcome 

TRCS 

outcome 

Parental  scales 

APQ — Poor  Monitoring  and  Supervision  Subscale 

APQ — Positive  Parenting  Subscale 

S 

>/ 

Child-Centered  Social  Control 

Confusion,  Hubbub,  and  Order 

S 

Community  Resources 

Community  Risk 

S 

Parent  and  teacher  Involvement 

Child  scales 

Altruistic  Behavior 

>/ 

Positive  Social  Behavior 

S 

Problem  Behavior 

S 

Teacher-reported 

Female 

S 

Hispanic  (non-White) 

S 

Black  (non-Hispanic) 

S 

Other  ethnicity 

S 

Total  teaching  experience 

S 

Total  experience  in  current  school 

Regular  certificate 

S 

Other  certificate 

Child  scales 

Academic  Competence  and  Motivation 

>/ 

ADHD-Related  Behavior 

/ 

>/ 

Altruistic  Behavior 

S 

>/ 

Positive  Social  Behavior 

y 

Problem  Behavior 

s 

Parent  and  Teacher  Involvement 

>/ 

NOTE:  Abbreviations  are 
CR:  Child  Report 
PCR:  Primary  Caregiver  Report 
TRS:  Teacher  Report  on  Student 
TRCS:  Teacher  Report  on  Classroom  and  School 
ADHD:  Attention  deficit  hyperactivity  disorder 
APQ:  Alabama  Parenting  Questionnaire 
V : Covariate  used 
Blank  cell:  Covariate  not  used 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Sample  Weights 

Sample  weights  were  used  in  all  analyses,  and  they  were  constructed  for  three  main  reasons:  (1)  to  give  each 
site  equal  weight  in  the  calculation  of  combined-program  impact  estimates,  (2)  to  give  each  school  equal  weight 
in  each  site,  and  (3)  to  adjust  for  missing  outcome  data  due  to  study  nonconsent  and  survey  nonresponse.32 
To  adjust  for  missing  student  data,  the  assumption  was  made  that  students  in  a specific  classroom  with  spring 
follow-up  data  were  representative  of  all  students  in  that  classroom.  Weights  were  constructed  to  be  inversely 
proportional  to  the  combined  consent  and  response  rates  within  each  classroom  (see  appendix  B).  These 
weights  were  constructed  separately  for  original  cohort  stayers  and  for  new  entrants  because  nonconsent  rates 
were  higher  for  the  new  entrants.  For  the  teacher-level  weights,  the  assumption  was  made  that  responding 
teachers  were  representative  of  all  third-,  fourth-,  and  fifth-grade  teachers  in  that  school.  The  standard  errors 
of  all  estimated  impacts  were  adjusted  for  design  effects  due  to  unequal  weighting. 

Statistical  Significance  and  Substantively  Important  Effects 

Results  are  provided  in  effect  sizes  that  were  calculated  by  dividing  the  estimated  impact  (the  coefficient 
estimated  by  the  regression  model)  by  the  standard  deviation  of  the  outcome  measure  for  the  control  group. 
The  standard  deviation  was  calculated  using  the  initial  data  for  the  weighted  control  group.  The  statistical 
significance  and  substantive  importance  of  each  result  were  identified.  A standard  two-tailed  test  was  used  to 
determine  the  Rvalue  for  the  impact  coefficient  of  each  outcome  measure.  Coefficients  with  ^-values  of  0.05 
or  below  were  considered  statistically  significant  and  identified  as  such.  In  addition,  coefficients  with  ^-values 
between  0.05  and  0.10  were  identified  in  the  tables  to  identify  where  there  may  have  been  additional 
significant  results  found  if  a larger  sample  had  been  used.  However,  these  results  were  not  considered  in  the 
text  or  compilations  of  statistically  significant  results. 

Individual  outcomes  having  non-statistically  significant  effect  sizes  of  0.25  or  above  (and  -0.25  or  below)  were 
identified  as  having  substantively  positive  (negative)  importance,  following  the  practice  used  by  the  What 
Works  Clearinghouse.  Substantive  importance  may  identify  impacts  of  practical  importance  that  might  have 
been  found  statistically  significant  if  the  sample  size  were  larger. 

In  addition  to  estimating  the  impacts  of  the  SACD  programs  on  the  individual  outcome  measures,  the 
impacts  on  the  four  domains  (Social  and  Emotional  Competence,  Behavior,  Academics,  and  Perceptions  of 
School  Climate)  were  also  examined.  Testing  the  impact  on  the  domains  was  done  to  adjust  for  the  multiple 
comparisons  made  within  each  domain  in  order  to  address  the  increased  chances  of  finding  a spurious 
outcome  when  more  than  one  test  was  done.  As  a result,  this  section  reports  two  sets  of  results:  (1)  the 
impacts  on  the  individual  outcomes  unadjusted  for  multiple  comparisons;  and  (2)  the  impacts  on  the  domains 
that  serve  as  the  multiple  comparison  adjustment. 

The  testing  of  the  significance  of  the  impacts  on  the  domains  was  based  on  a set  of  four  heuristics  that  were 
modeled  on  the  approach  used  by  the  U.S.  Department  of  Education’s  What  Works  Clearinghouse  for 
determining  whether  domain-level  effects  are  statistically  significant  when  there  are  multiple  outcome 
measures  within  a single  domain  (What  Works  Clearinghouse  n.d.-b).  The  first  three  of  these  heuristics  were 
also  applied  in  the  earlier  analysis  of  the  SACD  programs’  impacts  on  the  use  of  SACD  activities  to  six 
outcome  domains.33  Each  domain  was  checked  using  the  four  heuristics  and  a statistically  significant  positive 
impact  was  found  if  any  of  the  four  were  met.34  The  four  heuristics  were  as  follows: 


32  This  weighting  approach  produced  unbiased  estimates  of  the  average  treatment  effect  for  the  study  schools  and 
programs  that  were  purposively  selected  for  the  evaluation.  See  appendix  B for  more  information. 

33  The  fourth  heuristic  was  appropriate  for  use  with  the  student-level  outcomes  but  not  with  the  teacher-level  use  of 
SACD  activities  outcomes,  so  it  was  not  applied  to  the  six  domains  containing  the  individual  outcomes  of  SACD  activity 
use. 

34  For  ease  of  discussion,  this  section  describes  detecting  positive  significant  impacts.  These  heuristics  were  also  used  to 
detect  any  negative  impact  on  a domain. 
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1.  Based  on  the  results  from  the  statistical  test  of  each  outcome  variable  within  a domain  (unadjusted 
for  multiple  comparisons),  at  least  half  of  the  impacts  were  positive  and  statistically  significant  and  no 
impact  was  negative  and  statistically  significant. 

2.  The  omnibus  impact  for  all  the  outcomes  measured  together  was  positive  and  statistically  significant 
on  the  basis  of  a multivariate  statistical  test  (see  footnote  25). 

3.  After  applying  the  Benjamini-Hochberg  (Benjamini  and  Hochberg  1995)  procedure  to  each  outcome 
under  a domain,  at  least  one  of  the  outcomes  remained  positive  and  statistically  significant  and  no 
outcome  was  negative  and  statistically  significant.  The  Benjamini-Hochberg  procedure  adjusts 
significance  levels  downward  to  account  for  the  multiple  testing  of  impacts. 

4.  The  statistical  model  used  to  estimate  impacts  on  the  individual  outcomes  was  re-estimated  using  a 
composite  of  all  the  outcome  variables  under  a domain.  The  domain  was  found  significant  if  the 
impact  on  the  composite  was  significant.  The  composite  was  formed  by  standardizing  each  outcome 
variable  using  its  standard  deviation,  combining  the  values  of  the  outcome  variables,  and  taking  the 
average  of  the  final  value. 

Statistical  Power 

To  assess  the  statistical  power  of  the  combined-program  and  individual  program  impact  estimates  under  the 
SACD  design,  minimum  detectable  impacts  in  effect  size  (MDES)  units  for  each  outcome  measure  were 
calculated.  MDES  represent  the  smallest  impacts  in  effect  size  (standard  deviation)  units  that  can  be  detected 
with  a high  probability.  The  MDES  were  primarily  a function  of  study  sample  sizes,  the  degrees  of  freedom 
available  for  statistical  tests,  and  design  effects  due  to  clustering  (Schochet  2005).  Clustering  effects  are 
measured  by  intraclass  correlations  (ICCs)  that  reflect  the  percentage  of  the  total  variance  in  the  outcomes 
that  is  between  clusters  (schools).  These  MDES  were  calculated  at  80  percent  power.  Thus,  it  is  possible  to 
find  a statistically  significant  impact  estimate  on  an  outcome  when  the  true  impact  is  smaller  than  the  relevant 
MDES,  although  the  chance  that  this  will  occur  is  less  than  80  percent.  Similarly,  it  is  possible  to  find  an 
impact  estimate  that  is  not  statistically  significant  when  the  true  impact  is  as  large  as  the  MDES,  although  the 
chance  that  this  will  occur  is  20  percent  or  less. 

Table  1.25  displays  the  MDES  using  the  ICCs  from  equation  (2)  that  adjusted  them  using  the  covariates 
(assuming  a two-tailed  test  and  a 5 percent  significance  level).  For  the  outcomes  from  the  Child  Report,  the 
MDES  fell  below  0.1  except  for  Positive  School  Orientation  in  Years  2 and  3,  which  reached  about  0.13.  For 
outcomes  from  the  Primary  Caregiver  Report,  MDES  fell  below  0.05  for  all  3 years.  MDES  for  outcomes 
from  the  Teacher  Report  on  Student  ranged  from  0.05  to  0.24,  while  those  for  the  Teacher  Report  on 
Classroom  and  School  ranged  from  0.15  to  0.21. 

The  MDES  for  the  individual  programs  were  considerably  higher,  ranging  from  0.1  to  0.9  over  the  3 years, 
due  to  smaller  sample  sizes.  They  varied  somewhat  across  sites  with  no  consistent  pattern.  MDES  for  the 
specific  programs  can  be  found  in  the  program  chapters. 
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Table  1.25.  Adjusted  minimum  detectable  effect  sizes  for  combined-program  impact  evaluation 


Outcome  measure-Report 

Year  1 

Year  2 

Year  3 

Social  and  Emotional  Competence  Domain 

Self-Efficacy  for  Peer  Interaction-CR 

0.048 

0.048 

0.035 

Normative  Beliefs  About  Aggression-CR 

0.045 

0.081 

0.076 

Empathy-CR 

0.068 

0.078 

0.063 

Behavior  Domain 

Altruistic  Behavior-CR 

0.041 

0.057 

0.043 

Altruistic  Behavior-PCR 

0.035 

0.044 

0.047 

Altruistic  Behavior-TRS 

0.155 

0.238 

0.204 

Positive  Social  Behavior-PCR 

0.027 

0.042 

0.042 

Positive  Social  Behavior-TRS 

0.078 

0.115 

0.112 

Problem  Behavior-CR 

0.056 

0.086 

0.075 

Problem  Behavior-PCR 

0.047 

0.040 

0.046 

Problem  Behavior-TRS 

0.084 

0.086 

0.101 

ADHD-Related  Behavior-TRS 

0.071 

0.082 

0.101 

Academics  Domain 

Engagement  with  Learning-CR 

0.029 

0.036 

0.054 

Academic  Competence  and  Motivation-TRS 

0.053 

0.077 

0.097 

Perceptions  of  School  Climate  Domain 

Positive  School  Orientation-CR 

0.088 

0.123 

0.128 

Negative  School  Orientation-CR 

0.033 

0.070 

0.087 

Student  Afraid  at  School-CR 

0.048 

0.064 

0.059 

Victimization  at  School-CR 

0.046 

0.063 

0.065 

Feelings  of  Safety-TRCS 

0.171 

0.179 

0.207 

Student  Support  for  Teachers-TRCS 

0.151 

0.154 

0.188 

NOTE:  Abbreviations  are 
CR:  Child  Report 
PCR:  Primary  Caregiver  Report 
TRS:  Teacher  Report  on  Student 
TRCS:  Teacher  Report  on  Classroom  and  School 
ADHD:  Attention  deficit  hyperactivity  disorder 
The  minimum  detectable  effect  (MDE)  formula  used  in  the  calculations  is  as  follows: 

MDE  = f actor (df)  * \p1  (—  + — ) + (1  - A )(— — + — ' — ) 

\ sT  sc  sTnT  sciic 

where  s^and  sc  are  the  number  of  treatment  and  comparison  schools;  nT  and  nc  are  the  average  number  of  students  per  classroom: 
Pi  is  the  intraclass  correlation  (ICC)  at  the  school  level;  and  factor(df)  is  a constant  that  depends  on  the  number  of  degrees  of 
freedom  ( df)  available  for  analysis  (and  is  2.802  for  the  pooled  analysis).  Estimates  were  adjusted  for  fixed  program  effects  as  well  as 
baseline  covariates. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Results 

Table  1.26  provides  the  estimates  of  the  overall  impact  on  each  of  the  20  outcome  measures  over  each  of  the 
3 years.35  For  the  third-graders  in  Year  1 (using  the  spring  2005  data),  there  was  one  statistically  significant 
impact  of  the  SACD  programs,  when  combined,  on  the  teacher-reported  Student  Support  for  Teachers 
measure,  which  had  an  effect  size  of  0.12  standard  deviations.  Effect  sizes  ranged  from  0.00  to  0.12  (absolute 
value),  so  there  were  no  substantively  important  effects.  For  the  fourth-graders  in  Year  2 (using  spring  2006 
data),  there  was  one  statistically  significant  impact,  again  on  the  Student  Support  for  Teachers  measure,  which 
had  an  effect  size  of  0.16  standard  deviations.  The  other  effect  sizes  ranged  from  0.00  to  0.13  (absolute 
value),  and  there  were  no  substantively  important  effects.  For  the  fifth-graders  in  Year  3 (using  spring  2007 
data),  there  were  no  statistically  significant  impacts  of  the  SACD  programs  on  any  of  the  20  outcome 
measures.  Effect  sizes  ranged  from  0.00  to  0.08  (absolute  value),  and  there  were  no  substantively  important 
effects.  With  2 of  60  coefficients  found  statistically  significant  (less  than  the  3 that  would  be  expected  by 
chance),  and  none  found  substantively  important,  the  combined-program  evaluation  provides  no  support  for 
an  average  SACD  program  effect  on  student  outcomes.  Regarding  statistically  significant  impacts  on  the 
domains,  the  fourth  heuristic  indicated  a statistically  significant  detrimental  effect  of  the  SACD  programs  on 
the  domain  of  Social  and  Emotional  Competence  in  Years  2 and  3. 


35  Sample  sizes  are  not  reported  in  the  results  tables  because  they  vary  by  outcome  and  by  year.  Table  1.47  provides  the 
range  of  sample  sizes  for  the  outcomes  within  each  report  by  year. 
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Year  1 

(Spring  3rd  grade) 

Year  2 

(Spring  4th  grade) 

Year  3 

(Spring  5th  grade) 

Scale-Report 

T reat- 
ment 

Control 

Effect 
size  i 

o-value 

T reat- 
ment 

Control 

Effect 

size 

p-value 

T reat- 
ment 

Control 

Effect 

size 

p-value 

Social  and  Emotional  Competence  Domain1,2 

Self-Efficacy  for  Peer  Interactions-CR  (+) 

3.04 

3.06 

-0.03 

0.476 

3.18' 

' 3.22 

-0.07 

0.096 

3.22 

3.25 

-0.04 

0.233 

Normative  Beliefs  About  Aggression-CR  (-) 

1.29 

1.30 

-0.01 

0.826 

1.35 

1.36 

-0.02 

0.667 

1.46 

1.46 

-0.01 

0.913 

Empathy-CR  (+) 

2.32 

2.30 

0.06 

0.197 

2.18 

2.19 

-0.02 

0.739 

2.08 

2.10 

-0.05 

0.343 

Behavior  Domain 

Altruistic  Behavior-CR  (+) 

1.24A 

1.29 

-0.06 

0.094 

1.05 

1.09 

-0.06 

0.168 

1.04 

1.06 

-0.04 

0.368 

Altruistic  Behavior-PCR  (+) 

2.27A 

2.22 

0.07 

0.055 

2.24 

2.24 

0.00 

0.992 

2.25 

2.27 

-0.02 

0.691 

Altruistic  Behavior-TRS  (+) 

1.43 

1.39 

0.08 

0.380 

1.36 

1.35 

0.01 

0.935 

1.35 

1.37 

-0.04 

0.733 

Positive  Social  Behavior-PCR  (+) 

3.02 

3.03 

-0.01 

0.726 

3.08 

3.05 

0.05 

0.183 

3.10 

3.08 

0.03 

0.436 

Positive  Social  Behavior-TRS  (+) 

3.03 

3.02 

0.02 

0.659 

3.02 

3.04 

-0.02 

0.747 

3.10 

3.07 

0.03 

0.606 

Problem  Behavior-CR  (-) 

0.30 

0.30 

0.01 

0.849 

0.34 

0.35 

0.00 

0.940 

0.48 

0.47 

0.03 

0.518 

Problem  Behavior-PCR  (-) 

1.56 

1.56 

0.00 

0.914 

1.53 

1.54 

-0.03 

0.359 

1.53 

1.54 

-0.04 

0.335 

Problem  Behavior-TRS  (-) 

1.46 

1.45 

0.02 

0.620 

1.44 

1.45 

-0.02 

0.667 

1.45 

1.47 

-0.04 

0.467 

ADHD-Related  Behavior-TRS  (-) 

1.75 

1.75 

0.00 

0.925 

1.68 

1.70 

-0.04 

0.474 

1.65 

1.70 

-0.08 

0.205 

Academics  Domain 

Engagement  with  Learning-CR  (+) 

3.66 

3.68 

-0.04 

0.193 

3.67 

3.68 

-0.03 

0.462 

3.59 

3.61 

-0.04 

0.411 

Academic  Competence  and  Motivation-TRS  (+) 

2.98 

3.00 

-0.02 

0.353 

2.96 

2.98 

-0.02 

0.659 

2.98 

2.98 

0.00 

0.960 

See  notes  at  end  of  table. 
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Table  1.26.  Combined-program  impacts  on  outcomes — Continued 


Year  1 

(Spring  3rd  grade) 

Year  2 

(Spring  4th  grade) 

Year  3 

(Spring  5th  grade) 

Scale-Report 

T reat- 
ment 

Control 

Effect 

size 

p-value 

T reat- 
ment 

Control 

Effect 

size 

p-value 

T reat- 
ment 

Control 

Effect 

size 

p-value 

Perceptions  of  School  Climate  Domain 
Positive  School  Orientation-CR  (+) 

2.87 

2.85 

0.03 

0.573 

2.66 

2.64 

0.03 

0.717 

2.51 

2.56 

-0.07 

0.411 

Negative  School  Orientation-CR  (-) 

1.88 

1.91 

-0.04 

0.168 

1.96 

1.98 

-0.03 

0.466 

2.09 

2.10 

-0.01 

0.857 

Student  Afraid  at  School-CR  (-) 

2.26 

2.31 

-0.05 

0.165 

2.21 

2.26 

-0.06 

0.223 

2.20 

2.21 

-0.01 

0.818 

Victimization  at  School-CR  (-) 

0.76 

0.76 

-0.01 

0.760 

0.66 

0.69 

-0.04 

0.382 

0.72 

0.73 

-0.01 

0.913 

Feelings  of  Safety-TRCS  (+) 

3.47 

3.42 

0.06 

0.472 

3.47 

3.36 

0.13 

0.127 

3.31 

3.38 

-0.08 

0.445 

Student  Support  for  Teachers-TRCS  (+) 

3.50' 

' 3.39 

0.12 

0.046 

3.55' 

• 3.41 

0.16 

0.036 

3.43 

3.47 

-0.05 

0.546 

* Treatment  group  significantly  different  from  control  group  at  the  .05  level. 

A Treatment  group  significantly  different  from  control  group  at  the  .10  to  > .05  level. 

1 1mpact  on  domain  found  statistically  significant  and  detrimental  in  Year  2 based  on  the  fourth  heuristic,  in  which  the  statistical  model  used  to  estimate  impacts  on  the  individual 
outcomes  was  re-estimated  using  a composite  of  all  the  outcome  variables  under  a domain.  The  domain  was  found  significant  if  the  impact  on  the  composite  was  significant.  The 
composite  was  formed  by  standardizing  each  outcome  variable  using  its  standard  deviation,  combining  the  values  of  the  outcome  variables,  and  taking  the  average  of  the  final  value. 

2 Impact  on  domain  found  statistically  significant  and  detrimental  in  Year  3 based  on  the  fourth  heuristic. 

NOTE:  Abbreviations  are 
CR:  Child  Report 
PCR:  Primary  Caregiver  Report 
TRS:  Teacher  Report  on  Student 
TRCS:  Teacher  Report  on  Classroom  and  School 
ADHD:  Attention  deficit  hyperactivity  disorder 

The  +/-  signs  in  parentheses  indicate  the  direction  of  a beneficial  outcome.  All  impact  estimates  were  calculated  using  regression  models  where  each  program  and  school  within  a 
program  was  weighted  equally.  The  standard  errors  of  all  estimates  account  for  design  effects  due  to  unequal  weighting  and  the  clustering  of  students  within  schools.  The  effect  size 
was  calculated  by  dividing  the  estimated  impact  by  the  standard  deviation  of  the  outcome  measure  for  the  control  group.  Significance  is  based  on  p < .05.  See  table  1 .5  for  information 
about  the  measures  used  to  create  the  outcome  variables.  The  number  of  results  found  significant  was  no  more  than  expected  by  chance. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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The  lack  of  statistically  significant  beneficial  impact  estimates  at  the  overall  level  might  have  been  due  to 
beneficial  impacts  in  some  programs  that  were  offset  by  detrimental  impacts  in  others.  Estimated  impacts 
could  have  differed  across  programs  if  some  SACD  programs  were  more  effective  than  others,  or  if  the  types 
of  children  or  settings  in  some  programs  were  more  conducive  to  positive  program  effects  than  others.  To 
investigate  this  possibility,  differences  in  estimated  impacts  across  programs  were  tested,  as  was  the  statistical 
significance  of  program-specific  impact  estimates. 

Impact  estimates,  however,  did  not  differ  significantly  across  programs.  Using  an  F test  to  test  for  differences 
in  estimated  impacts  across  programs  led  to  the  finding  that  none  of  the  differences  in  pooled  impact 
estimates  across  programs  was  statistically  significant  at  the  5 percent  level  in  any  of  the  3 years.  This  result 
provided  evidence  that  it  was  not  the  case  that  one  or  two  programs  drove  or  masked  the  estimated  impacts 
in  the  combined  data. 

Table  1.27  lists  the  outcomes  on  which  each  individual  program  had  statistically  significant  impacts  in  each 
year  (a  program’s  impacts  on  all  the  outcomes  are  given  in  the  program-specific  chapters).  There  were  5 
significant  program  impacts  (3  beneficial  and  2 detrimental)  in  Year  1,  7 (5  beneficial  and  2 detrimental)  in 
Year  2,  and  4 (1  beneficial  and  3 detrimental)  in  Year  3,  for  a total  of  16  (9  beneficial  and  7 detrimental) 
program-level  impacts.  As  a comparison,  7 of  the  140  comparisons  made  for  each  year  would  be  expected  to 
be  statistically  significant  by  chance,  which  would  produce  a total  of  21  expected  significant  impacts.  Table 
1.27  also  lists  the  19  nonsignificant  substantive  impacts  by  program;  10  were  beneficial  and  9 were 
detrimental. 
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Table  1.27.  Individual  program  statistically  significant  impacts  and  nonsignificant  but  substantively  important  impacts 


Statistically  significant1 Nonstatistically  significant  but  substantive2 


Program 

Beneficial  impacts 
(Report)  (Effect  size)  (p-value) 

Detrimental  impacts 
(Report)  (Effect  size)  (p-value) 

Beneficial  impacts 
(Report)  (Effect  size)  (p-value) 

Detrimental  impacts 
(Report)  (Effect  size)  (p-value) 

Total 

9 

7 

10 

9 

All  programs 
Year  1 

3 

2 

2 

0 

Year  2 

5 

2 

6 

1 

Year  3 

1 

3 

2 

8 

ABC 
Year  1 

Altruistic  Behavior 
(TRS)  (.39)  (.026) 

Year  2 

Academic  Competence 
(CR)  (.31)  (.011) 
Feelings  of  Safety 
(TRCS)  (.75)  (.003) 

Altruistic  Behavior 
(CR)  (-.20)  (0.029) 

Student  Support  for  Teachers  (TRCS) 
(.27)  (.276) 

Year  3 

Positive  Social  Behavior 
(PCR)  (.21)  (.041) 

Feelings  of  Safety 
(TRCS)  (.31)  (.235) 

CSP 
Year  1 

Year  2 

Problem  Behavior 
(PCR)  (-.21)  (.042) 

Altruistic  Behavior 
(TRS)  (.47)  (.132) 
Student  Afraid  at  School 
(CR)  (-.26)  (.090) 

Year  3 

Altruistic  Behavior 
(TRS)  (-.41)  (.132) 
Feelings  of  Safety 
(TRCS)  (-.36)  (.246) 

LBW 
Year  1 

Altruistic  Behavior 
(PCR)  (.31)  (.005) 
Student  Support  for  Teachers 
(TRCS)  (.52)  (.022) 

Year  2 

Student  Support  for  Teachers 
(TRCS)  (.28)  (.428) 

Altruistic  Behavior 
(TRS)  (-.34)  (.270) 

See  notes  at  end  of  table. 
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Table  1.27.  Individual  program  statistically  significant  impacts  and  nonsignificant  but  substantively  important  impacts— Continued 


Statistically  significant1 Nonstatistically  significant  but  substantive2 

Beneficial  impacts  Detrimental  impacts  Beneficial  impacts  Detrimental  impacts 


Program  (Report)  (Effect  size)  (p-value)  (Report)  (Effect  size)  (p-value)  (Report)  (Effect  size)  (p-value)  (Report)  (Effect  size)  (p-value) 


Year  3 

Engagement  with  Learning 
(CR)  (-.35)  (.030) 
Positive  School  Orientation 
(CR)  (-.33)  (.047) 
Feelings  of  Safety 
(TRCS)  (-.70)  (.046) 

Problem  Behavior 
(CR)  (.31)  (.223) 
Student  Support  for  Teachers 
(TRCS)  (-.26)  (.543) 

PA 

Year  1 
Year  2 

Year  3 

Engagement  with  Learning 
(CR)  (-.25)  (.017) 

Altruistic  Behavior 
(TRS)  (.27)  (.480) 

Positive  Social  Behavior 
(PCR)  (.24)  (.039) 
Problem  Behavior 
(TRS)  (-.24)  (.048) 

Student  Support  for  Teachers 
(TRCS)  (.28)  (.113) 

PATHS 
Year  1 
Year  2 
Year  3 

Altruistic  Behavior 
(TRS)  (-.31)  (.485) 
Feelings  of  Safety 
(TRS)  (-.29)  (.582) 

4Rs 
Year  1 

Year  2 
Year  3 

Academic  Competence 
(CR)  (-.17)  (.032) 

Feelings  of  Safety 
(TRS)  (-.42)  (1.46) 
Student  Support  for  Teachers 
(TRCS)  (-.35)  (.109) 

See  notes  at  end  of  table. 
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Table  1.27.  Individual  program  statistically  significant  impacts  and  nonsignificant  but  substantively  important  impacts— Continued 


Statistically  significant1 Nonstatistically  significant  but  substantive2 

Beneficial  impacts  Detrimental  impacts  Beneficial  impacts  Detrimental  impacts 


Program  (Report)  (Effect  size)  (p-value)  (Report)  (Effect  size)  (p-value)  (Report)  (Effect  size)  (p-value)  (Report)  (Effect  size)  (p-value) 


SS 

Year  1 
Year  2 
Year  3 

Feelings  of  Safety 
(TRCS)  (.37)  (.216) 

Positive  Social  Behavior 
(PCR)  (-.14)  (.050) 

Feelings  of  Safety 
(TRCS)  (.39)  (.197) 

Feelings  of  Safety 
(TRCS)  (.52)  (.062) 

1 Of  the  1 40  comparisons  made  for  each  year,  7 would  be  expected  to  be  statistically  significant  at  the  .05  level  by  chance  (for  a total  of  21 ). 

2 Defined  as  impacts  that  were  not  statistically  significant  but  were  .25  standard  deviation  units  (absolute  value)  or  more  in  magnitude. 
NOTE:  Abbreviations  are 


ABC:  Academic  and  Behavioral  Competencies  Program 
CSP:  Competence  Support  Program 
LBW:  Love  In  a Big  World 
PA:  Positive  Action 

o PATHS:  Promoting  Alternative  Thinking  Strategies 

4Rs:  The  4Rs  Program  (Reading,  Writing,  Respecting,  and  Resolution) 

SS:  Second  Step 

CR:  Child  Report 

PCR:  Primary  Caregiver  Report 

TRS:  Teacher  Report  on  Student 

TRCS:  Teacher  Report  on  Classroom  and  School 

Blank  cell:  Finding  of  no  impact. 

All  impact  estimates  were  calculated  using  regression  models  in  which  each  school  within  a program  was  weighted  equally.  The  standard  errors  of  all  estimates  account  for  design 
effects  due  to  unequal  weighting  and  the  clustering  of  students  within  schools.  Significance  is  based  on  p < .05.  The  number  of  results  found  significant  was  no  more  than  expected  by 
chance. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Sensitivity  Analyses 

To  estimate  the  multilevel  models,  it  was  necessary  to  make  decisions  about  key  model  parameter 
specifications  and  estimation  methods.  These  decisions  had  the  potential  to  affect  the  results.  A set  of 
sensitivity  analyses  was  done  to  determine  if  these  model  assumptions  affected  the  results  from  the 
combined-program  impact  analysis.  Ten  sensitivity  tests  were  done,  and  these  included  the  following 
variations: 

1.  The  models  were  estimated  without  the  initial  covariates. 

2.  The  models  were  estimated  without  sample  weights  and  also  with  weights  that  were  not  adjusted  for 
nonconsent  and  nonresponse. 

3.  Classroom-level  random  effects  were  included  in  the  error  structure. 

4.  The  pairwise  matching  of  schools  was  accounted  for  in  the  error  structure. 

5.  The  pretests  were  treated  as  dependent  variables  rather  than  as  covariates. 

6.  The  models  were  estimated  using  alternative  statistical  software  packages. 

7.  Missing  outcome  measures  were  imputed  using  multiple  imputation  procedures. 

8.  Combined-program  impact  estimates  were  obtained  by  averaging  the  program-level  impact  estimates. 

9.  Restricted  sets  of  covariates  were  included  in  the  models. 

10.  New  entrants  were  excluded  from  the  analysis. 

The  number  of  sensitivity  analyses  done  each  year  declined  as  the  pattern  of  impact  results  from  the  original 
model  proved  robust  to  a variety  of  model  specifications.  All  10  sensitivity  analyses  were  done  with  the  Year 
1 data.  The  first  nine  sensitivity  tests  were  done  with  the  Year  2 data.  The  sample  of  new  entrants  grew  large 
enough  in  Years  2 and  3 to  do  separate  analyses  of  new  entrants  versus  stayers  (in  place  of  the  10th  sensitivity 
test)  and  these  were  included  in  the  subgroup  analyses  discussed  in  chapter  1.  The  first,  second  (except  for 
the  estimation  without  weights),  third,  and  fourth  sensitivity  tests  were  done  using  the  Year  3 data. 

The  results  from  the  sensitivity  analyses  were  similar  to  those  of  the  original  combined-program  analysis  and 
did  not  produce  additional  evidence  of  impacts  on  the  outcome  measures.  Of  the  10  sensitivity  analyses,  5 
found  no  significant  impacts  on  the  20  outcomes,  4 found  one  significant  impact,  and  1 found  two  significant 
impacts.  The  sensitivity  analyses  and  their  results  are  discussed  in  appendix  B. 

Year-by-Year  Analysis  of  Impacts  on  Student  Subgroups 

The  social  and  character  development  evaluation  research  questions  included  the  expectation  that  the  SACD 
programs,  considered  together,  might  be  more  effective  for  some  groups  of  children  than  others.  These 
subgroups  were  based  on  (1)  student  gender;  (2)  student  risk  factors;  (3)  student  status  as  a stayer  (member  of 
the  original  sample  from  fall  2004)  or  a new  entrant  to  the  study;  and  (4)  fidelity  of  implementation  in  the 
student’s  school.  Descriptive  analyses,  discussed  earlier,  of  the  initial  data  confirmed  that  the  characteristics 
used  to  form  the  first  two  subgroups  were  randomly  distributed  between  the  treatment  and  control  groups, 
and  found  no  statistically  significant  difference  in  the  composition  of  the  treatment  and  control  groups  by 
stayers  and  new  entrants.  These  subgroups  were  hypothesized  to  respond  differently  to  the  SACD  programs 
based  on  (1)  indications  in  the  literature  that  gender,  socioeconomic  risk,  family  risk,  and  community  risk  are 
associated  with  children’s  behavioral  and  academic  outcomes;  and  (2)  results  from  previous  evaluations  of 
universal  school-based  SACD  interventions,  which  have  found  that  impacts  can  differ  for  children  at  high 
risk  for  behavior  problems  and  academic  failure,  and  that  impacts  can  be  greater  for  children  who  have 
received  greater  program  exposure. 

The  identification  of  gender  and  of  stayer  versus  new  entrant  status  was  straightforward  because  these  were 
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clearly  distinctive  subgroups  of  boys  versus  girls  and  students  who  were  in  the  sample  from  the  initial  data 
collection  versus  those  who  joined  afterward.  Identification  of  the  subgroups  defined  by  initial  risk  and 
fidelity  of  implementation  required  the  use  of  scales. 

To  identify  students  with  initial  risks,  four  domains  of  initial  risk  were  examined: 

1 . Socioeconomic  risk  was  measured  by  the  number  of  the  following  risk  factors  the  child  faced  at  the 
beginning  of  the  evaluation:  (1)  the  child  was  in  a single-parent  household,  (2)  the  child  was  in  a 
low -income  household  (below  135  percent  of  the  federal  poverty  level),  and  (3)  the  child’s 
primary  caregiver  was  a high  school  dropout.  The  risk  measure  used  in  the  analysis  was  a 
cumulative  score  across  the  three  risk  variables  with  the  following  values:  0 (no  risk  factors),  1 
(one  risk  factor),  and  2 (two  or  three  risk  factors). 

2.  Family  risk  was  defined  as  poor  parental  supervision  and  taken  from  the  Alabama:  Poor  Supervision 
Scale  from  the  Primary  Caregiver  Report  data. 

3.  Perceived  community  risk  was  defined  as  perceived  neighborhood  risk  measured  through  items 
from  the  Community  Risk  Scale  from  the  Primary  Caregiver  Report  data. 

4.  Child  behavior  risk  was  measured  by  the  BASC  Conduct  Problems  Subscale,  responses  to  which 
were  collected  in  both  the  Teacher  Report  on  Student  and  the  Primary  Caregiver  Report. 

Thus,  there  were  five  risk  measures  used  to  examine  the  four  initial  risk  domains:  (1)  one  measure  for 
socioeconomic  risk,  which  consisted  of  a scale  based  on  three  variables;  (2)  one  measure  for  family  risk, 
which  was  based  on  one  variable;  (3)  one  measure  for  community  risk,  which  was  based  on  one  variable;  and 
(4)  two  measures  for  child  behavior  risk.  Other  variables  were  considered  for  each  risk  measure  (e.g.,  parents’ 
employment  status  for  socioeconomic  risk,  positive  parenting  for  family  risk,  and  community  resources  for 
community  risk)  but,  in  a regression  analysis  of  the  initial  data,  these  variables  were  not  found  to  predict  the 
outcome  measures.  The  five  selected  risk  measures  were  treated  in  the  analysis  as  continuous,  rather  than 
categorical,  variables  to  preserve  all  risk  information,  and  because  there  were  no  well-established  rules  for 
defining  risk  cutoffs  and  groups  for  which  the  effects  of  the  SACD  evaluation  were  likely  to  differ. 

Fidelity  of  implementation  of  the  SACD  programs  was  measured  differently  in  each  of  the  three  years.  In 
Year  1,  treatment  schools  were  divided  into  two  fidelity  groups:  (1)  high  implementers  (20  schools)  and 
(2)  low  implementers  (22  schools).  In  Years  2 and  3,  schools  were  again  divided  into  the  high  (20  schools  in 
Year  2,  19  schools  in  Year  3)  and  low  implementers  (22  schools  in  Year  2,  23  schools  in  Year  3)  but  the 
fidelity  measure  combined  the  rankings  from  multiple  years  into  three  categories:  (1)  high  implementers  for 
all  years,  (2)  low  implementers  for  all  years,  and  (3)  mixed  implementers  (high  and  low  implementers  in 
different  years).  The  discussion  of  the  construction  of  the  fidelity  of  implementation  variables  noted  that  the 
fidelity  analysis  was  considered  exploratory,  both  because  of  how  the  variable  was  constructed  and  because  of 
the  nonrandom  assignment  of  high  and  low  fidelity.  As  a result,  differences  in  estimated  impacts  by  fidelity 
group  could  reflect  differences  in  the  level  of  implementation  and/or  underlying  differences  between  the 
students,  the  schools,  and  the  SACD  programs. 

Analysis 

The  subgroup  analyses  were  conducted  at  the  combined -program  level.  The  1 8 student-level  outcomes  were 
used.  The  degree  of  power  available  for  testing  the  differences  between  the  subgroups  was  lower  than  for  the 
whole-group  analysis  and  depended  on  the  number  and  sizes  of  the  subgroups.  Therefore  the  results  were 
measured  less  precisely  than  for  the  year-by-year  impacts  at  the  combined-program  level.  Partly  as  a result  of 
this,  and  because  the  What  Works  Clearinghouse  heuristics  do  not  specify  how  to  conduct  multiple 
comparison  tests  across  subgroups,  the  heuristics  were  not  applied  to  examine  subgroup  impact  results  by 
domain.36 


36  The  comparison  approach  used  was  to  assess  whether  there  were  differences  in  impacts  across  subgroups  (e.g., 
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Two  of  the  subgroups  were  discrete  binary  variables  (boy/girl  or  stayer/new  entrant)  as  was  Year  1 fidelity 
(high/low),  one  subgroup  included  three  discrete  binary  variables  (fidelity  for  Years  2 and  3),  and  the  initial 
risk  variables  were  continuous  (and  so  were  not  truly  discrete  subgroups).  For  all  the  subgroup  analyses,  the 
model  used  to  test  the  equality  of  the  subgroup  impacts  can  be  represented  in  a unified  equation  as 


Yisp  — Yo  + YlTsp  ~h  dp  ~h  Xisp/3  +GispA  ~hGispTspP  ~h  [lisp  ~h  G is/) ] 

where  Yisp  is  an  outcome  for  a student;  Tsp  is  a binary  variable  equal  to  1 for  treatment  group  schools  and  0 
for  control  schools;  0P  are  program-specific  fixed  effects  coefficients  to  be  estimated;  Xisp  are  student-  or 
teacher-level  baseline  covariates  that  do  not  include  the  subgroup  of  interest;  Gisp  is  the  variable  to  represent 
the  subgroup  of  interest;  [3,  ya,  yi,  K and  p are  fixed  parameter  vectors  to  be  estimated;  eisp  are  assumed  to  be 
independent  and  identically  distributed  N(0,o2e)  student-  or  teacher-level  random  error  terms;  and  usp  are 
independent  and  identically  distributed  N(0,a2u)  school-specific  error  terms  that  capture  the  correlation 
between  the  outcomes  of  students  in  the  same  schools  and  are  assumed  to  be  distributed  independently  of 
the  eisp  error  terms.  When  subgroups  are  discrete,  G;sp  is  a binary  variable  for  the  nonexcluded  subgroup  of 
interest  (such  as  females,  when  males  are  the  excluded  subgroup).37  For  the  risk  variables  that  were 
continuously  defined,  Gisp  is  a continuous  variable. 

In  this  model,  p is  the  coefficient  on  the  interaction  term  between  the  subgroup  of  interest  and  the  treatment 
group  indicator  variable.  The  value  of  p estimates  the  additional  effect  on  the  impact  estimate  of  switching 
subgroups  (through  a one-unit  change  in  the  Gisp  variable).  A test  of  the  hypothesis  that  p equals  0 is 
conducted  to  determine  whether  the  subgroups  have  statistically  significantly  different  impacts. 

This  equation  is  the  same  for  the  subgroup  analyses  in  which  there  are  only  two  subgroups  and  for  the 
subgroup  analyses  that  use  a continuous  variable.  In  the  case  of  three  subgroups  (such  as  with  the  three-tiered 
fidelity  categorization),  Gisp  includes  two  subgroups  (because  the  third  one  is  excluded),  and  there  are  two 
subgroup-by-treatment-status  interaction  terms.  The  joint  test  of  whether  the  coefficients  on  these  terms 
equal  0 is  conducted  to  determine  whether  the  three  subgroups  have  different  impacts.  For  the  discrete  binary 
variables,  p represents  the  additional  effect  of  changing  from  boy  to  girl  or  from  stayer  to  new  entrant.  For 
continuous  variables,  p represents  the  additional  effect  of  a one-unit  change  in  the  Gisp  variable.  For 
presentation  purposes,  because  one -unit  changes  in  the  risk  measures  might  be  hard  to  interpret  because  the 
risk  measures  are  scales,  the  point  estimates  of  the  impacts  are  presented  at  the  mean  risk  level,  one  standard 
deviation  above  the  mean  risk  level,  and  one  standard  deviation  below  the  mean  risk  level. 

The  subgroup  analyses  determined  if  there  was  a differential  impact  of  the  SACD  programs  on  the 
subgroups.  In  cases  where  such  a difference  was  found,  a secondary  analysis  was  done  to  obtain  more 
information  on  this  difference  by  testing  the  statistical  significance  of  the  impact  estimates  for  each  subgroup 
separately.  These  supplemental  analyses  could  help  identify  why  a differential  impact  occurred.  For  example, 
if  a significant  differential  impact  was  found  that  favored  girls,  the  supplemental  analysis  might  determine  that 
the  difference  was  due  to  either  a beneficial  impact  on  girls  or  a detrimental  impact  on  boys.38  Impacts  for 


impacts  for  boys  versus  girls)  for  each  outcome  and  then  to  examine  impacts  on  the  individual  subgroups  (e.g.,  boys). 
The  latter  were  to  be  examined  only  when  the  former  were  found  significant.  The  alternative  approach  of  applying 
multiple  comparisons  corrections  for  all  the  outcomes  under  each  subgroup  was  not  applied  because  it  would  have 
reduced  the  alpha  levels  for  significance  testing  to  such  low  values  (e.g.,  from  .05  to  .005)  that  there  would  be  little 
statistical  power  to  detect  impacts. 

37  For  the  subgroup  analyses,  (1)  girls  were  included  and  boys  excluded,  (2)  new  entrants  were  included  and  stayers 
excluded,  and  (3)  low  fidelity  was  excluded. 

38  The  student  gains  for  some  of  the  outcomes  are  represented  by  positive  coefficients  (an  increase  in  that  outcome  is 
desired)  while  for  others  a gain  is  represented  by  a negative  coefficient  (a  decrease  is  desired).  To  reduce  confusion, 
impacts  are  described  as  beneficial  or  detrimental  for  a subgroup.  A beneficial  outcome  means  that  a desired  outcome 
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discrete  (binary  or  categorical)  subgroups  were  estimated  using  the  equation  (2)  regression  models  (described 
in  Year-by-Year  Impacts  on  Students  and  Perceptions  of  School  Climate)  separately  for  each  subgroup — for 
example,  separately  for  boys  and  for  girls.  For  the  continuous  initial  risk  variables,  equation  (2)  was  estimated 
three  times  for  each  risk  variable:  once  for  each  level  of  risk  reported  (mean  risk  level,  one  standard  deviation 
above  the  mean  risk  level,  and  one  standard  deviation  below  the  mean  risk  level). 

It  is  possible  to  have  a statistically  significant  difference  in  estimated  impacts  between  subgroups  and  not  to 
have  statistically  significant  estimated  impacts  for  one  or  all  of  the  subgroups.  In  a similar  way,  there  can  be 
statistically  significant  estimated  impacts  for  the  subgroups,  but  no  statistically  significant  difference  between 
these  impacts.  These  patterns  can  occur  because  the  tests  of  statistical  significance  are  asking  slightly  different 
questions.  For  each  subgroup-specific  regression,  the  hypothesis  that  is  being  tested  is  whether  the  impact 
coefficient  is  statistically  significantly  different  from  zero,  when  the  coefficients  for  the  regressor  variables  are 
fit  specifically  for  that  subgroup.  For  the  test  of  the  significance  of  the  difference  in  the  impact  estimates,  the 
hypothesis  is  testing  whether  the  difference  in  the  impact  estimates  for  each  subgroup  is  statistically 
significantly  different  from  zero,  when  the  coefficients  for  the  regressor  variables  are  fit  for  the  full  sample 
(all  subgroups  combined). 

Results 

For  each  subgroup,  the  results  of  the  primary  and  supplemental  subgroup  tests  are  discussed.  The  test  of  the 
significance  of  the  difference  in  the  impact  estimates  for  each  subgroup  is  discussed  first.  This  test  directly 
compares  the  point  estimates  of  the  impacts  for  each  level  of  the  subgroup,  while  taking  into  account  the 
variance  around  each  point  estimate.  Formally,  it  is  a test  of  the  null  hypothesis  that  the  difference  in  impact 
between  subgroup  levels  equals  zero.  The  test  establishes  whether  differences  in  impacts  between  levels  of 
the  subgrouping  variable  exist.  Second,  the  results  of  the  / tests  from  the  subgroup-specific  regressions  are 
discussed.  These  tests  determine  the  significance  of  the  impact  of  the  intervention  on  each  of  the  subgroups. 
The  findings  from  the  subgroup-specific  regressions  are  discussed  only  if  the  test  of  the  significance  of  the 
difference  in  the  impact  estimates  results  rejects  the  hypothesis  that  the  difference  between  the  impacts  for 
the  subgroups  is  zero  (although  for  completeness,  the  tables  present  all  the  subgroup-specific  regression 
results). 

Tables  1.28  through  1.35  present  the  year -by-year  results  for  the  subgroup  analyses.  In  the  tables,  the  ^-value 
for  the  difference  in  the  impact  estimates  for  each  subgroup  is  labeled  “p-Value  for  Difference”  for  the 
discrete  subgroups  (gender,  stayer/ new  entrant,  and  fidelity  of  implementation)  and  “Marginal  Effect”  for  the 
initial  risk  subgroups.  The  “Marginal  Effect”  column  contains  both  the  p-v alue  and  the  effect  of  moving  from 
one  level  of  risk  up  to  the  next  level.39  The  other  columns  give  the  coefficients  converted  into  effect  sizes  and 
the  /)-values  from  the  subgroup-specific  regressions,  and  are  labeled  by  their  subgroup  (e.g.,  “Boys”  and 
“Girls”  in  table  1.28). 

Gender 

There  were  8 cases  of  significant  differences  between  the  impacts  of  the  seven  SACD  programs  on  boys 
versus  girls  (table  1.28,  last  column  under  each  year).  From  the  54  comparisons  made,  3 significant 
differences  would  be  expected  by  chance.  When  examining  the  results  of  the  / tests  from  subgroup-specific 
regressions  for  these  8 cases,  1 case  was  based  on  a significant  beneficial  program  impact  for  boys  (an 
increase  in  Positive  Social  Behavior  in  Year  2);  3 were  based  on  significant  detrimental  program  impacts  for 
boys  (declines  in  Altruistic  Behavior  in  Years  1 and  2,  and  Engagement  with  Learning  in  Year  2);  1 was  based 


has  increased  or  an  undesired  outcome  has  decreased  for  that  subgroup.  A detrimental  outcome  means  that  a desired 
outcome  has  decreased  or  an  undesired  outcome  has  increased  for  that  subgroup. 

39  The  marginal  effects  for  the  discrete  binary  subgroups  (gender  and  stayer/ new  entrant)  can  be  approximated  by 
subtracting  the  effect  sizes  from  the  subgroup-specific  regressions  because  the  differences  between  subgroups  in 
standard  deviation  estimates  are  small.  For  the  fidelity  subgroups,  this  approach  can  be  used  to  identify  pairwise 
differences  between  the  impacts  for  the  low-,  mixed-,  and  high-fidelity  subgroups. 
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on  a significant  beneficial  impact  for  girls  (a  decline  in  Negative  School  Orientation  in  Year  2);  and  1 was 
based  on  a significant  detrimental  impact  for  girls  (a  decline  in  Academic  Competence  in  Year  1).  Of  the 
remaining  2,  the  first  showed  more  beneficial  impacts  for  girls  (a  gain  in  Normative  Beliefs  About  Aggression 
in  Year  2)  while  the  second  showed  more  beneficial  impacts  for  boys  (a  decline  in  ADHD-related  behavior  in 
Year  3)  but  neither  was  linked  to  a significant  program  impact  on  either  gender.  Because  there  were  few 
significant  differences  and  they  did  not  provide  consistent  results  (e.g.,  half  showed  a detrimental  program 
impact),  these  results  provide  little  evidence  that  the  SACD  programs,  as  a group,  differentially  improved 
outcomes  for  one  gender. 
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Table  1.28.  Combined-program  impacts  on  child  outcomes,  by  gender 


Year  1 Year  2 Year  3 

(Spring  3rd  grade) (Spring  4th  grade) (Spring  5th  grade) 


Boys 

Girls 

p-value 

for 

Boys 

Girls 

p-value 

for 

Boys 

Girls 

p-value 

for 

Scale-Report 

Effect 

size 

P- 

value 

Effect 

size 

P- 

value 

differ- 

ence1 

Effect 

size 

P- 

value 

Effect 

size 

P- 

value 

differ- 

ence1 

Effect 

size 

P- 

value 

Effect 

size 

P- 

value 

differ- 

ence1 

Social  and  Emotional 
Competence  Domain 
Self-Efficacy  for  Peer 
Interactions-CR  (+) 

0.01 

0.907 

-0.05 

0.345 

0.558 

-0.1 0A 

0.076 

-0.04 

0.479 

0.281 

-0.09A 

0.083 

0.01 

0.833 

0.092' 

Normative  Beliefs 
About  Aggression- 
CR(-) 

-0.01 

0.842 

-0.01 

0.899 

0.788 

0.04 

0.574 

-0.1 0A 

0.095 

0.023* 

-0.04 

0.545 

0.02 

0.768 

0.245 

Empathy-CR  (+) 

0.04 

0.500 

0.08 

0.133 

0.705 

-0.05 

0.401 

0.02 

0.685 

0.165 

-0.06 

0.248 

-0.03 

0.661 

0.841 

Behavior  Domain 
Altruistic  Behavior- 
CR  (+) 

-0.13* 

0.014 

0.01 

0.921 

0.023* 

-0.13* 

0.020 

0.00 

0.967 

0.022* 

-0.04 

0.453 

-0.03 

0.580 

0.784 

Altruistic  Behavior- 
PCR  (+) 

0.07 

0.199 

0.09A 

0.081 

0.754 

-0.07 

0.173 

0.07 

0.177 

0.063A 

-0.10 

0.102 

0.05 

0.343 

0.101 

Altruistic  Behavior- 
TRS  (+) 

0.04 

0.667 

0.09 

0.398 

0.330 

0.01 

0.924 

0.02 

0.918 

0.972 

-0.01 

0.940 

-0.03 

0.785 

0.981 

Positive  Social 
Behavior-PCR  (+) 

0.00 

0.934 

-0.01 

0.790 

0.904 

0.10* 

0.032 

-0.02 

0.653 

0.033* 

0.04 

0.512 

0.00 

0.985 

0.826 

Positive  Social 
Behavior-TRS  (+) 

0.01 

0.755 

0.01 

0.750 

0.993 

-0.02 

0.830 

-0.02 

0.769 

0.978 

0.10 

0.200 

-0.01 

0.864 

0.058' 

Problem  Behavior- 
CR  (-) 

0.01 

0.897 

0.00 

0.993 

0.927 

-0.01 

0.852 

-0.01 

0.900 

0.810 

0.03 

0.675 

0.02 

0.735 

0.990 

Problem  Behavior- 
PCR  (-) 

0.02 

0.630 

-0.01 

0.875 

0.851 

-0.08 

0.138 

0.03 

0.599 

0.089A 

-0.03 

0.697 

-0.04 

0.493 

0.895 

Problem  Behavior- 
TRS  (-) 

0.02 

0.715 

0.03 

0.577 

0.865 

-0.02 

0.720 

-0.03 

0.601 

0.788 

-0.08 

0.346 

-0.03 

0.569 

0.621 

ADHD-Related 
Behavior-TRS  (-) 

-0.03 

0.518 

0.02 

0.619 

0.254 

-0.06 

0.346 

-0.02 

0.745 

0.529 

-0.1 5A 

0.064 

-0.02 

0.736 

0.043’ 

See  notes  at  end  of  table. 
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Table  1.28.  Combined-program  impacts  on  child  outcomes,  by  gender— Continued 


Year  1 Year  2 Year  3 

(Spring  3rd  grade) (Spring  4th  grade) (Spring  5th  grade) 


Boys 

Girls 

p-value 

for 

Boys 

Girls 

p-value 

for 

Boys 

Girls 

p-value 

for 

Scale-Report 

Effect 

size 

P- 

value 

Effect 

size 

P- 

value 

differ- 

ence1 

Effect 

size 

P- 

value 

Effect 

size 

P- 

value 

differ- 

ence1 

Effect 

size 

P- 

value 

Effect 

size 

P- 

value 

differ- 

ence1 

Academics  Domain 
Engagement  with 
Learning-CR  (+) 

-0.01 

0.872 

-0.08 

0.133 

0.178 

-0.10* 

0.050 

0.05 

0.297 

0.011* 

0.00 

0.998 

-0.09 

0.100 

0.317 

Academic 
Competence  and 
Motivation-TRS  (+) 

0.04 

0.331 

-0.07* 

0.017 

0.003* 

0.02 

0.684 

-0.04 

0.424 

0.242 

0.05 

0.386 

-0.05 

0.443 

0.093' 

Perceptions  of  School 
Climate  Domain 
Positive  School 
Orientation-CR  (+) 

0.04 

0.529 

0.02 

0.728 

0.593 

-0.01 

0.872 

0.06 

0.393 

0.081 A 

-0.09 

0.291 

-0.04 

0.617 

0.345 

Negative  School 
Orientation-CR  (-) 

0.01 

0.815 

-0.10* 

0.040 

0.065A 

0.09 

0.126 

-0.14* 

0.011 

0.000* 

-0.01 

0.914 

0.00 

0.943 

0.842 

Student  Afraid  at 
School-CR  (-) 

-0.02 

0.654 

-0.08 

0.102 

0.299 

-0.10 

0.102 

-0.02 

0.716 

0.214 

0.01 

0.818 

-0.02 

0.666 

0.445 

Victimization  at 
School-CR  (-) 

-0.02 

0.748 

-0.01 

0.817 

0.983 

-0.07 

0.223 

-0.01 

0.876 

0.262 

-0.01 

0.850 

0.01 

0.809 

0.683 

* Treatment  group  significantly  different  from  control  group  at  the  .05  level. 

A Significantly  different  from  zero  at  the  .10  to  > .05  level. 

1 From  the  test  of  a significant  difference  between  Boys’  effect  size  and  Girls’  effect  size. 

NOTE:  Abbreviations  are 
CR:  Child  Report 
PCR:  Primary  Caregiver  Report 
TRS:  Teacher  Report  on  Student 
ADHD:  Attention  deficit  hyperactivity  disorder 

The  +/-  signs  in  parentheses  indicate  the  direction  of  a beneficial  outcome.  All  impact  estimates  were  calculated  using  regression  models  in  which  each  program  and  school  within  a 
program  was  weighted  equally.  The  standard  errors  of  all  estimates  account  for  design  effects  due  to  unequal  weighting  and  the  clustering  of  students  within  schools.  The  effect  size 
was  calculated  by  dividing  the  estimated  impact  by  the  standard  deviation  of  the  outcome  measure  for  the  control  group. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Stayer  Versus  New  Entrant 

For  Years  2 and  3,  no  significant  differences  between  the  impacts  on  stayers  versus  new  entrants  were  found 
(table  1.29).  From  the  36  comparisons  made,  2 outcomes  would  have  been  expected  to  be  statistically 
significant  by  chance.  These  results  provide  litde  evidence  that  the  SACD  programs,  as  a group,  differentially 
improved  outcomes  for  stayers  or  for  new  entrants. 

The  results  comparing  new  entrants  and  stayers  subgroups  might  have  been  sensitive  to  the  methods  used  to 
fill  in  missing  values  in  the  initial  data  for  new  entrants.  This  could  be  a concern,  especially  because  time- 
varying  characteristics  were  measured  at  the  start  of  the  evaluation  for  stayers,  but  no  data  were  available  for 
new  entrants.  Information  collected  during  the  follow-up  periods  was  used  to  fill  in  missing  values  for  initial 
data  items  that  were  not  expected  to  vary  much  over  time.  For  characteristics  that  were  likely  to  vary 
considerably  over  time,  including  initial  values  of  the  outcome  measures,  values  were  imputed  based  on  cell 
means,  where  cells  were  defined  by  students’  schools,  gender,  and  race/ethnicity.  To  check  the  sensitivity  of 
these  results  to  the  imputation  of  covariates  that  might  vary  over  time,  results  were  estimated  for  the  stayer 
and  new  entrant  subgroups  using  only  covariates  that  were  unlikely  to  vary  over  time.  The  subgroup  results 
were  similar  (these  data  are  not  shown  in  a table). 
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Table  1.29.  Combined-program  impacts  on  child  outcomes,  by  stayer  versus  new  entrant 


Year  2 

Year  3 

(Spring  4th  grade) 

(Spring  5th  grade) 

Stayers 

New  entrants 

p-value 

for 

Stayers 

New  entrants 

p-value 

for 

Effect 

Effect 

Effect 

Effect 

Scale-Report 

size 

p-value 

size 

p-value 

difference1 

size 

p-value 

size 

p-value 

difference1 

Social  and  Emotional  Competence  Domain 

Self-Efficacy  for  Peer  Interactions-CR  (+) 

-0.06 

0.176 

-0.14 

0.117 

0.319 

-0.04 

0.326 

-0.10 

0.184 

0.723 

Normative  Beliefs  About  Aggression-CR  (-) 

0.00 

0.988 

-0.11 

0.136 

0.140 

0.00 

0.957 

-0.05 

0.491 

0.542 

Empathy-CR  (+) 

-0.01 

0.798 

-0.04 

0.566 

0.785 

-0.05 

0.353 

0.01 

0.847 

0.794 

Behavior  Domain 

Altruistic  Behavior-CR  (+) 

-0.03 

0.516 

-0.14A 

0.066 

0.132 

-0.03 

0.496 

0.02 

0.832 

0.966 

Altruistic  Behavior-PCR  (+) 

-0.01 

0.866 

0.02 

0.820 

0.618 

-0.04 

0.443 

0.01 

0.950 

0.246 

Altruistic  Behavior-TRS  (+) 

0.02 

0.903 

-0.01 

0.971 

0.793 

-0.02 

0.868 

-0.05 

0.676 

0.420 

Positive  Social  Behavior-PCR  (+) 

0.01 

0.813 

0.1 3A 

0.093 

0.067A 

0.02 

0.691 

0.02 

0.825 

0.444 

Positive  Social  Behavior-TRS  (+) 

-0.02 

0.818 

-0.06 

0.462 

0.680 

0.05 

0.460 

-0.03 

0.679 

0.361 

Problem  Behavior-CR  (-) 

-0.01 

0.871 

0.02 

0.827 

0.473 

0.01 

0.852 

0.02 

0.747 

0.380 

Problem  Behavior-PCR  (-) 

-0.05 

0.244 

-0.11 

0.160 

0.195 

-0.05 

0.426 

-0.04 

0.610 

0.634 

Problem  Behavior-TRS  (-) 

-0.04 

0.486 

0.04 

0.648 

0.370 

-0.07 

0.293 

0.02 

0.826 

0.353 

ADHD-Related  Behavior-TRS  (-) 

-0.03 

0.590 

-0.01 

0.880 

0.913 

-0.09 

0.121 

0.01 

0.873 

0.333 

Academics  Domain 

Engagement  with  Learning-CR  (+) 

-0.03 

0.533 

-0.06 

0.396 

0.773 

-0.05 

0.331 

0.01 

0.876 

0.443 

Academic  Competence  and  Motivation-TRS  (+) 

-0.07 

0.170 

-0.05 

0.494 

0.645 

-0.02 

0.739 

-0.07 

0.363 

0.275 

See  notes  at  end  of  table. 
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Table  1.29.  Combined-program  impacts  on  child  outcomes,  by  stayer  versus  new  entrant— Continued 


Year  2 

(Spring  4th  grade) 

Year  3 

(Spring  5th  grade) 

Stayers 

New  entrants 

Stayers 

New  entrants 

Scale-Report 

Effect 

size  p-value 

Effect 

size 

p-value 

p-value  for 
difference1 

Effect 

size 

p-value 

Effect 

size 

p-value 

p-value  for 
difference1 

Perceptions  of  School  Climate  Domain 
Positive  School  Orientation-CR  (+) 

0.05  0.561 

0.05 

0.595 

0.225 

-0.05 

0.526 

-0.07 

0.430 

0.559 

Negative  School  Orientation-CR  (-) 

-0.05  0.432 

0.08 

0.280 

0.053A 

-0.04 

0.493 

0.05 

0.524 

0.230 

Student  Afraid  at  School-CR  (-) 

-0.05  0.418 

-0.01 

0.889 

0.676 

0.02 

0.717 

-0.02 

0.824 

0.329 

Victimization  at  School-CR  (-) 

-0.05  0.353 

-0.04 

0.626 

0.622 

0.01 

0.845 

0.02 

0.830 

0.663 

A Significantly  different  from  zero  at  the  .10  to  > .05  level. 

1 From  the  test  of  a significant  difference  between  Stayers’  effect  size  and  New  Entrants’  effect  size. 


NOTE:  Abbreviations  are 
CR:  Child  Report 
PCR:  Primary  Caregiver  Report 
TRS:  Teacher  Report  on  Student 
ADHD:  Attention  deficit  hyperactivity  disorder 

g The  +/-  signs  in  parentheses  indicate  the  direction  of  a beneficial  outcome.  No  findings  were  found  statistically  significant  at  or  below  the  .05  level.  All  impact  estimates  were  calculated 

o using  regression  models  in  which  each  program  and  school  within  a program  was  weighted  equally.  The  standard  errors  of  all  estimates  account  for  design  effects  due  to  unequal 

weighting  and  the  clustering  of  students  within  schools.  The  effect  size  was  calculated  by  dividing  the  estimated  impact  by  the  standard  deviation  of  the  outcome  measure  for  the 
SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Initial  Risk 

The  impacts  of  the  seven  SACD  programs,  considered  together,  on  initial  student  risk  were  examined  using 
five  risk  measures  (one  measure  each  for  socioeconomic  risk,  family  risk,  and  perceptions  of  community  risk, 
and  two  measures  for  child  behavior  risk).  The  significance  of  the  change  in  the  impact  estimate  over  a one- 
unit  change  (moving  one  level  between  the  three  risk  levels)  in  the  risk  measure  (termed  the  marginal  effect ) was 
tested  (see  tables  1.30  to  1.34).  A total  of  270  marginal  effects  were  tested  over  the  3 years  (90  per  year),  and 
13  to  14  would  be  expected  to  be  significant  by  chance.  A total  of  41  were  found  significant — 13  from  Year 
1,  17  from  Year  2,  and  11  from  Year  3.  Of  these  41  significant  findings,  26  showed  a more  beneficial 
marginal  impact  for  high-risk  students  and  15  showed  a more  detrimental  impact  for  high-risk  students 
compared  to  lower  risk  students.  Six  of  these  findings  were  repeated  in  at  least  2 different  years,  with  1 of 
these  repeated  all  3 years  (4  showing  a more  beneficial  marginal  effect  for  high-risk  students  and  2 showing  a 
detrimental  impact). 

For  24  of  the  41  significant  marginal  effects,  there  were  no  significant  findings  from  the  subgroup -specific 
regressions.  In  12  of  the  41  significant  marginal  effects,  the  subgroup-specific  regressions  supported  the 
interpretation  of  the  marginal  effect  (6  results  supported  a more  beneficial  impact  for  high-risk  students  and  6 
results  supported  a more  detrimental  impact  for  high-risk  students).  In  the  remaining  5 significant  marginal 
effects,  the  subgroup-specific  regressions  supported  a different  interpretation:  (1)  a detrimental  effect  on 
lower  risk  students  rather  than  a beneficial  impact  on  higher  risk  students  in  two  cases,  and  (2)  a beneficial 
effect  on  lower  risk  students  rather  than  a detrimental  impact  on  higher  risk  students  in  three  cases. 

The  low  number,  mixed  direction  (beneficial  and  detrimental),  and  lack  of  repeated  findings  of  the  significant 
results  provide  litde  evidence  of  a relationship  between  risk  levels  and  the  impact  of  the  seven  SACD 
programs  considered  together.  Discussions  of  the  results  for  each  type  of  risk  follow. 

Socioeconomic  Risk 

For  initial  socioeconomic  risk,  there  were  5 significant  marginal  effects,  all  showing  detrimental  impacts  on 
high-risk  students.  From  the  54  comparisons  made,  3 significant  differences  would  be  expected  by  chance. 
Table  1.30  presents  the  impacts  of  the  seven  programs  together  on  the  outcome  measures  by  level  of  student 
socioeconomic  risk.  An  example  may  help  in  the  interpretation  of  the  table.  For  the  outcome  Engagement 
with  Learning  under  the  Academics  domain,  in  Year  1 the  estimated  marginal  effect  of  -0.131  indicates  that, 
as  the  risk  level  increased  one  unit,  the  estimated  impact  of  the  seven  SACD  programs  together  decreased  by 
0.131.  This  estimate  of  the  marginal  effect  was  statistically  significantly  different  from  zero,  providing 
evidence  that  the  estimated  impact  of  the  seven  SACD  programs,  as  a group,  on  this  outcome  varied  across 
socioeconomic  risk  levels  (in  this  case  it  was  detrimental  as  Engagement  with  Learning  declined  as  risk 
increased).  The  results  from  the  subgroup-specific  regressions  are  provided  at  three  levels  of  initial  risk: 
(1)  low,  which  is  one  standard  deviation  below  the  mean  socioeconomic  risk  level;  (2)  average,  which  is  the 
mean  risk  level;  and  (3)  high,  which  is  one  standard  deviation  above  the  mean  risk  level.  In  Year  1,  at  the  low 
level  of  socioeconomic  risk,  the  estimated  impact  was  0.041  and  nonsignificant.  As  the  level  of 
socioeconomic  risk  increased  to  the  mean  level,  with  all  else  held  equal,  the  point  estimate  for  the  impact 
estimate  changed  to  a nonsignificant  —0.049.  As  the  level  of  socioeconomic  risk  continued  to  increase  to  one 
standard  deviation  above  the  mean  level,  all  else  held  equal,  the  estimated  impact  changed  to  -0.154.  The 
point  estimate  for  the  impact  at  the  high  level  of  risk  was  statistically  significantly  different  from  zero, 
providing  additional  evidence  that  the  seven  SACD  programs,  together,  had  a detrimental  impact  on 
Engagement  with  Learning  for  students  at  high  socioeconomic  risk.  However,  as  these  results  were  not 
repeated  for  Years  2 and  3,  no  clear  pattern  of  impact  on  this  outcome  emerged. 

In  addition  to  a reduction  in  Engagement  with  Learning,  the  high-risk  students  experienced  a detrimental 
program  impact  on  four  other  outcomes:  declines  in  Altruistic  Behavior  (Teacher  Report  in  Year  1)  and 
Positive  School  Orientation  (Year  1)  and  gains  in  Negative  School  Orientation  (Year  1)  and  Student  Afraid  at 
School  (Year  3).  None  of  the  significant  marginal  effects  occurred  for  more  than  1 year. 


110 


Table  1.30.  Combined-program  impacts  on  child  outcomes,  by  initial  socioeconomic  risk 

Year  1 Year  2 Year  3 

(Spring  3rd  grade) (Spring  4th  grade) (Spring  5th  grade) 


Effect  size  at  risk  level 
(p-value) 

Marg- 

inal 

Effect  size  at  risk  level 
(p-value) 

Marg- 

inal 

Effect  size  at  risk  level 
(p-value) 

Marg- 

inal 

Scale-Report 

Low1 

Aver- 

age2 

High3 

effect4 

(p-value) 

Low1 

Aver- 

age2 

High3 

effect4 

(p-value) 

Low1 

Aver- 

age2 

High3 

effect4 

(p-value) 

Social  and  Emotional 
Competence  Domain 

Self-Efficacy  for  Peer 

-0.005 

-0.027 

-0.052 

-0.032 

-0.075 

-0.077A 

-0.079 

-0.003 

-0.065 

-0.042 

-0.015 

0.038 

Interactions-CR  (+) 

(0.922) 

(0.490) 

(0.298) 

(0.433) 

(0.165) 

(0.089) 

(0.161) 

(0.955) 

(0.190) 

(0.284) 

(0.779) 

(0.439) 

Normative  Beliefs 

-0.024 

0.001 

0.026 

0.033 

-0.002 

-0.009 

-0.018 

-0.012 

0.047 

0.032 

0.015 

-0.024 

About  Aggression-CR  (-) 

(0.655) 

(0.982) 

(0.642) 

(0.449) 

(0.976) 

(0.881) 

(0.801) 

(0.810) 

(0.479) 

(0.576) 

(0.822) 

(0.637) 

Empathy-CR  (+) 

0.124* 

0.077 

-0.022 

-0.069 

-0.009 

-0.027 

-0.048 

-0.031 

-0.019 

-0.068 

-0.126A 

-0.081 

(0.038) 

(0.129) 

(0.713) 

(0.110) 

(0.891) 

(0.632) 

(0.457) 

(0.533) 

(0.777) 

(0.255) 

(0.073) 

(0.122) 

Behavior  Domain 

Altruistic  Behavior- 

-0.012 

-0.054 

-0.102* 

-0.060 

-0.081 

-0.067 

-0.051 

0.024 

-0.073 

-0.031 

0.018 

0.069 

CR  (+) 

(0.800) 

(0.169) 

(0.044) 

(0.138) 

(0.139) 

(0.143) 

(0.375) 

(0.628) 

(0.171) 

(0.470) 

(0.744) 

(0.168) 

Altruistic  Behavior- 

0.047 

0.032 

0.014 

-0.023 

-0.008 

0.006 

0.022 

0.023 

0.010 

-0.013 

-0.040 

-0.038 

PCR  (+) 

(0.322) 

(0.396) 

(0.781) 

(0.590) 

(0.884) 

(0.900) 

(0.697) 

(0.643) 

(0.882) 

(0.811) 

(0.549) 

(0.495) 

Altruistic  Behavior- 

0.164 

0.085 

-0.006 

-0.115* 

0.035 

0.014 

-0.012 

-0.037 

-0.030 

-0.052 

-0.078 

-0.037 

TRS  (+) 

(0.102) 

(0.365) 

(0.950) 

(0.011) 

(0.827) 

(0.931) 

(0.940) 

(0.492) 

(0.808) 

(0.661) 

(0.526) 

(0.457) 

Positive  Social 

-0.012 

-0.026 

-0.041 

-0.020 

0.025 

0.042 

0.062 

0.029 

0.027 

0.036 

0.045 

0.014 

Behavior-PCR  (+) 

(0.751) 

(0.394) 

(0.298) 

(0.555) 

(0.603) 

(0.285) 

(0.207) 

(0.509) 

(0.619) 

(0.429) 

(0.425) 

(0.786) 

Positive  Social 

0.020 

-0.002 

-0.027 

-0.031 

0.002 

-0.032 

-0.072 

-0.058 

0.058 

0.009 

-0.050 

-0.082' 

Behavior-TRS  (+) 

(0.693) 

(0.968) 

(0.594) 

(0.315) 

(0.979) 

(0.670) 

(0.370) 

(0.212) 

(0.459) 

(0.906) 

(0.530) 

(0.089) 

Problem  Behavior- 

-0.029 

-0.001 

0.031 

0.040 

0.000 

-0.002 

-0.006 

-0.005 

0.028 

0.073 

0.124 

0.073 

CR  (-) 

(0.620) 

(0.983) 

(0.597) 

(0.318) 

(0.997) 

(0.971) 

(0.940) 

(0.925) 

(0.708) 

(0.293) 

(0.108) 

(0.151) 

Problem  Behavior- 

-0.015 

0.010 

0.039 

0.037 

0.005 

-0.013 

-0.034 

-0.030 

-0.070 

-0.034 

0.009 

0.061 

PCR  (-) 

(0.752) 

(0.801) 

(0.423) 

(0.327) 

(0.910) 

(0.743) 

(0.493) 

(0.509) 

(0.230) 

(0.478) 

(0.877) 

(0.267) 

Problem  Behavior- 

-0.009 

0.052 

0.1 01 A 

0.062A 

-0.004 

-0.006 

-0.009 

-0.004 

-0.043 

-0.034 

-0.024 

0.014 

TRS  (-) 

(0.871) 

(0.293) 

(0.066) 

(0.056) 

(0.950) 

(0.914) 

(0.891) 

(0.931) 

(0.572) 

(0.621) 

(0.752) 

(0.768) 

ADHD-Related 

0.022 

0.020 

0.018 

-0.003 

-0.004 

-0.019 

-0.038 

-0.026 

-0.053 

-0.067 

-0.084 

-0.024 

Behavior-TRS  (-) 

(0.645) 

(0.635) 

(0.716) 

(0.923) 

(0.953) 

(0.724) 

(0.546) 

(0.565) 

(0.459) 

(0.297) 

(0.243) 

(0.615) 

See  notes  at  end  of  table. 
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Table  1.30.  Combined-program  impacts  on  child  outcomes,  by  initial  socioeconomic  risk— Continued 


Year  1 

(Spring  3rd  grade) 

Effect  size  at  risk  level 
(p-value) 


Scale-Report 

Academics  Domain 


Marg- 

inal 

effect4 

>-value) 


Year  2 

(Spring  4th  grade) 

Effect  size  at  risk  level 
(p-value) 


Marg- 

inal 

effect4 

r-value) 


Year  3 

(Spring  5th  grade) 

Effect  size  at  risk  level 
(p-value) 


Engagement  with 
Learning-CR  (+) 

0.041 

(0.394) 

-0.049 

(0.202) 

-0.154** 

(0.003) 

-0.131** 

(0.003) 

Academic  Competence  and 
Motivation-TRS  (+) 

-0.013 

(0.688) 

-0.002 

(0.941) 

-0.011 

(0.743) 

0.017 

(0.532) 

Perceptions  of  School 
Climate  Domain 

Positive  School 
Orientation-CR  (+) 

0.092 

(0.237) 

0.011 

(0.876) 

-0.083 

(0.286) 

-0.118** 

(0.009) 

Negative  School 
Orientation-CR  (-) 

-0.103A 

(0.057) 

-0.045 

(0.319) 

0.023 

(0.672) 

0.085* 

(0.043) 

Student  Afraid  at 
School-CR  (-) 

-0.088 

(0.139) 

-0.058 

(0.242) 

-0.024 

(0.688) 

0.043 

(0.331) 

Victimization  at 
School-CR  (-) 

-0.027 

(0.600) 

-0.009 

(0.835) 

0.012 

(0.814) 

0.026 

(0.528) 

-0.037 

-0.030 

-0.021 

0.012 

-0.041 

-0.046 

-0.051 

-0.008 

(0.421) 

(0.412) 

(0.668) 

(0.796) 

(0.460) 

(0.318) 

(0.377) 

(0.882) 

-0.014 

-0.016 

-0.018 

-0.003 

0.029 

0.019 

0.008 

-0.016 

(0.760) 

(0.687) 

(0.703) 

(0.936) 

(0.622) 

(0.713) 

(0.899) 

(0.707) 

0.058 

0.028 

-0.008 

-0.051 

-0.049 

-0.090 

-0.140 

-0.070 

(0.509) 

(0.732) 

(0.930) 

(0.315) 

(0.609) 

(0.311) 

(0.146) 

(0.178) 

-0.019 

-0.013 

-0.007 

0.009 

-0.024 

0.000 

0.029 

0.040 

(0.783) 

(0.827) 

(0.922) 

(0.852) 

(0.745) 

(0.996) 

(0.699) 

(0.427) 

-0.043 

-0.049 

-0.056 

-0.010 

-0.081 

-0.019 

0.054 

0.103 

(0.522) 

(0.410) 

(0.417) 

(0.849) 

(0.224) 

(0.743) 

(0.427) 

(0.047) 

-0.029 

-0.048 

-0.071 

-0.033 

0.012 

0.014 

0.017 

0.004 

(0.603) 

(0.303) 

(0.216) 

(0.503) 

(0.838) 

(0.771) 

(0.777) 

(0.938) 

* Significantly  different  from  zero  at  the  .05  level. 

**  Significantly  different  from  zero  at  the  .01  level. 

A Significantly  different  from  zero  at  the  .10  to  > .05  level. 

1 One  standard  deviation  below  the  mean  risk  level. 

2 At  the  mean  risk  level. 

3 One  standard  deviation  above  the  mean  risk  level. 

4 Change  in  impact  as  risk  level  increases  by  one  unit. 

NOTE:  Abbreviations  are 

CR:  Child  Report 

PCR:  Primary  Caregiver  Report 

TRS:  Teacher  Report  on  Student 

ADHD:  Attention  deficit  hyperactivity  disorder 

The  +/-  signs  in  parentheses  indicate  the  direction  of  a beneficial  outcome.  All  impact  estimates  were  calculated  using  regression  models  in  which  each  program  and  school  within 
a program  was  weighted  equally.  The  standard  errors  of  all  estimates  account  for  design  effects  due  to  unequal  weighting  and  the  clustering  of  students  within  schools.  The  effect 
size  was  calculated  by  dividing  the  estimated  impact  by  the  standard  deviation  of  the  outcome  measure  for  the  control  group. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Family  Risk 

Over  the  3 years,  there  were  6 statistically  significant  marginal  effects  regarding  level  of  family  risk  (table 
1.31).  From  the  54  comparisons  made,  3 significant  differences  would  be  expected  by  chance.  Five  of  the 
significant  results  showed  beneficial  impacts  for  high-risk  students:  increases  in  Empathy  (Year  1), 
Engagement  with  Learning  (Year  3),  and  Academic  Competence  (Year  3),  and  declines  in  Problem  Behavior 
(Child  Report  in  Year  3)  and  Negative  School  Orientation  (Year  3).  The  results  from  the  subgroup-specific 
regressions  provided  support  for  the  Empathy  finding  but  offered  an  alternative  explanation  for  the 
Engagement  with  Learning  finding,  as  they  suggested  that  the  significant  marginal  effect  was  due  to  a 
detrimental  effect  on  low-risk  students  rather  than  a beneficial  impact  on  high-risk  students. 

The  sixth  significant  marginal  effect  showed  a detrimental  impact  on  high-risk  students  of  an  increase  in 
Student  Afraid  at  School  (Year  2).  The  subgroup-specific  regression  results  suggested  that  this  significant 
marginal  effect  was  due  to  a beneficial  effect  on  low-risk  students  rather  than  a detrimental  impact  on  high- 
risk  students. 
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Table  1.31.  Combined-program  impacts  on  child  outcomes, 


Year  1 

(Spring  3rd  grade) 

Effect  size  at  risk  level 

Marg- 

(p-value) ina| 

Aver-  effect4 

Scale-Report Low1  age2  High3  (p-value) 

Social  and  Emotional 
Competence  Domain 


Self-Efficacy  for  Peer 

0.007 

-0.031 

-0.068 

-0.169 

Interactions-CR  (+) 

(0.890) 

(0.434) 

(0.170) 

(0.215) 

Normative  Beliefs 

0.036 

-0.008 

-0.052 

-0.198 

About  Aggression-CR  (-) 

(0.532) 

(0.859) 

(0.359) 

(0.178) 

Empathy-CR  (+) 

0.000 

0.073 

0.146* 

0.329* 

(0.996) 

(0.171) 

(0.019) 

(0.018) 

Behavior  Domain 

Altruistic  Behavior-CR  (+) 

-0.124* 

-0.067 

-0.010 

0.258A 

(0.016) 

(0.104) 

(0.847) 

(0.057) 

Altruistic  Behavior-PCR  (+) 

0.036 

0.033 

0.029 

-0.017 

(0.476) 

(0.416) 

(0.565) 

(0.905) 

Altruistic  Behavior-TRS  (+) 

0.095 

0.080 

0.064 

-0.069 

(0.353) 

(0.414) 

(0.527) 

(0.609) 

Positive  Social 

-0.053 

-0.025 

0.003 

0.125 

Behavior-PCR  (+) 

(0.199) 

(0.439) 

(0.940) 

(0.267) 

Positive  Social 

-0.043 

-0.005 

0.033 

0.1 71 A 

Behavior-TRS  (+) 

(0.407) 

(0.920) 

(0.515) 

(0.077) 

Problem  Behavior-CR  (-) 

0.051 

0.004 

-0.060 

-0.251 A 

(0.392) 

(0.933) 

(0.314) 

(0.053) 

Problem  Behavior-PCR  (-) 

-0.007 

0.012 

0.032 

0.089 

(0.871) 

(0.740) 

(0.481) 

(0.459) 

Problem  Behavior-TRS  (-) 

0.090 

0.053 

-0.016 

-0.1 68A 

(0.108) 

(0.301) 

(0.780) 

(0.097) 

ADHD-Related 

0.026 

0.016 

-0.060 

-0.045 

Behavior-TRS  (-) 

(0.593) 

(0.714) 

(0.907) 

(0.648) 

See  notes  at  end  of  table. 


initial  family  risk 


Year  2 


Year  3 


(Spring  4th  grade) (Spring  5th  grade) 


Effect  size  at  risk  level 

Marg- 

Effect  size  at  risk  level 

Marg- 

(p-value) 

inal 

(p-value) 

inal 

Aver- 

effect4 

Aver- 

effect4 

Low1  age2  High3 

(p-value) 

Low1  age2  High3 

(p-value) 

-0.063 

-0.052 

-0.041 

0.049 

-0.059 

-0.020 

0.019 

0.175 

(0.278) 

(0.261) 

(0.470) 

(0.751) 

(0.309) 

(0.641) 

(0.749) 

(0.313) 

0.034 

-0.003 

-0.040 

-0.165 

0.072 

-0.002 

-0.076 

-0.333' 

(0.690) 

(0.969) 

(0.635) 

(0.301) 

(0.396) 

(0.974) 

(0.362) 

(0.056) 

-0.056 

-0.014 

0.028 

0.191 

-0.084 

-0.066 

-0.049 

0.079 

(0.400) 

(0.807) 

(0.667) 

(0.213) 

(0.277) 

(0.318) 

(0.523) 

(0.650) 

-0.100 

-0.039 

0.022 

0.275A 

-0.068 

-0.032 

0.003 

0.160 

(0.109) 

(0.446) 

(0.714) 

(0.081) 

(0.299) 

(0.532) 

(0.960) 

(0.367) 

0.021 

0.000 

-0.022 

-0.097 

-0.029 

-0.041 

-0.053 

-0.053 

(0.720) 

(0.996) 

(0.707) 

(0.532) 

(0.694) 

(0.505) 

(0.469) 

(0.767) 

0.016 

0.026 

0.036 

0.045 

-0.093 

-0.022 

0.049 

0.321  ■ 

(0.925) 

(0.878) 

(0.836) 

(0.783) 

(0.480) 

(0.863) 

(0.706) 

(0.053) 

-0.054 

-0.005 

0.043 

0.218A 

-0.041 

0.010 

0.060 

0.226 

(0.279) 

(0.891) 

(0.373) 

(0.094) 

(0.542) 

(0.865) 

(0.364) 

(0.146) 

-0.064 

-0.037 

-0.009 

0.125 

-0.012 

0.030 

0.072 

0.190 

(0.460) 

(0.653) 

(0.917) 

(0.354) 

(0.887) 

(0.712) 

(0.409) 

(0.209) 

0.018 

-0.025 

-0.069 

-0.196 

0.141 

0.034 

-0.073 

-0.483’ 

(0.827) 

(0.737) 

(0.402) 

(0.182) 

(0.106) 

(0.666) 

(0.398) 

(0.004) 

0.002 

0.033 

0.065 

0.142 

0.013 

-0.019 

-0.052 

-0.147 

(0.974) 

(0.397) 

(0.187) 

(0.299) 

(0.842) 

(0.720) 

(0.426) 

(0.377) 

0.031 

-0.013 

-0.057 

-0.198 

0.026 

-0.033 

-0.093 

-0.267' 

(0.677) 

(0.854) 

(0.448) 

(0.142) 

(0.770) 

(0.685) 

(0.297) 

(0.082) 

-0.007 

-0.029 

-0.051 

-0.100 

-0.079 

-0.091 

-0.103 

-0.053 

(0.919) 

(0.636) 

(0.452) 

(0.460) 

(0.301) 

(0.184) 

(0.177) 

(0.730) 

Chapter  1.  The  Social  and  Character  Development  Multiprogram  Evaluation 


Table  1.31. 


Combined-program  impacts  on  child  outcomes,  by  initial  family  risk— Continued 


Year  1 

Year  2 

Year  3 

(Spring  3rd  grade) 

(Spring  4th  grade) 

(Spring  5th  grade) 

Effect  size  at  risk  level 

Marg- 

Effect size  at  risk  level 

Marg- 

Effect size  at  risk  level 

Marg- 

(p-value) 

inal 

(p-value) 

inal 

(p-value) 

inal 

Aver- 

effect4 

Aver- 

effect4 

Aver- 

effect4 

Scale-Report 

Low1 

age2 

High3 

(p-value) 

Low1 

age2 

High3 

(p-value) 

Low1 

age2 

High3 

(p-value) 

Academics  Domain 

Engagement  with 

-0.090A 

-0.056 

-0.022 

0.155 

-0.081 

-0.013 

0.054 

0.304A 

-0.152* 

-0.067 

0.018 

0.382* 

Learning-CR  (+) 

(0.086) 

(0.170) 

(0.676) 

(0.294) 

(0.159) 

(0.765) 

(0.338) 

(0.060) 

(0.025) 

(0.206) 

(0.790) 

(0.040) 

Academic  Competence 

-0.013 

-0.004 

0.020 

0.075 

-0.043 

-0.011 

0.020 

0.140 

-0.069 

0.003 

0.075 

0.324* 

and  Motivation-TRS  (+) 

(0.694) 

(0.893) 

(0.540) 

(0.386) 

(0.387) 

(0.784) 

(0.686) 

(0.218) 

(0.291) 

(0.953) 

(0.243) 

(0.015) 

Perceptions  of  School 
Climate  Domain 

Positive  School 

0.007 

0.006 

0.006 

-0.003 

0.002 

0.040 

0.079 

0.173 

-0.115 

-0.085 

-0.054 

0.136 

Orientation-CR  (+) 

(0.931) 

(0.931) 

(0.941) 

(0.985) 

(0.983) 

(0.647) 

(0.402) 

(0.256) 

(0.245) 

(0.353) 

(0.579) 

(0.414) 

Negative  School 

-0.065 

-0.044 

-0.023 

0.096 

0.000 

-0.051 

-0.101 

-0.226 

0.052 

-0.051 

-0.1 53A 

-0.461** 

Orientation-CR  (-) 

(0.272) 

(0.389) 

(0.699) 

(0.478) 

(0.997) 

(0.489) 

(0.211) 

(0.140) 

(0.505) 

(0.461) 

(0.051) 

(0.006) 

Student  Afraid  at 

-0.096 

-0.058 

-0.020 

0.170 

-0.175* 

-0.084 

0.006 

0.408* 

-0.001 

0.003 

0.007 

0.018 

School-CR  (-) 

(0.113) 

(0.257) 

(0.735) 

(0.228) 

(0.020) 

(0.192) 

(0.932) 

(0.012) 

(0.989) 

(0.968) 

(0.933) 

(0.918) 

Victimization  at 

-0.055 

-0.004 

0.048 

0.232A 

-0.030 

-0.042 

-0.054 

-0.054 

0.036 

0.008 

-0.021 

-0.128 

School-CR  (-) 

(0.298) 

(0.935) 

(0.361) 

(0.090) 

(0.622) 

(0.396) 

(0.367) 

(0.727) 

(0.604) 

(0.894) 

(0.766) 

(0.451) 

* Significantly  different  from  zero  at  the  .05  level. 


**  Significantly  different  from  zero  at  the  .01  level. 

A Significantly  different  from  zero  at  the  .10  to  > .05  level. 

1 One  standard  deviation  below  the  mean  risk  level. 

2 At  the  mean  risk  level. 

3 One  standard  deviation  above  the  mean  risk  level. 

4 Change  in  impact  as  risk  level  increases  by  one  unit. 

NOTE:  Abbreviations  are 

CR:  Child  Report 

PCR:  Primary  Caregiver  Report 

TRS:  Teacher  Report  on  Student 

ADHD:  Attention  deficit  hyperactivity  disorder 

The  +/-  signs  in  parentheses  indicate  the  direction  of  a beneficial  outcome.  All  impact  estimates  were  calculated  using  regression  models  in  which  each  program  and  school  within  a 
program  was  weighted  equally.  The  standard  errors  of  all  estimates  account  for  design  effects  due  to  unequal  weighting  and  the  clustering  of  students  within  schools.  The  effect  size 
was  calculated  by  dividing  the  estimated  impact  by  the  standard  deviation  of  the  outcome  measure  for  the  control  group. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Perceptions  of  Community  Risk. 

Over  the  3 years,  there  were  1 0 significant  marginal  effects  regarding  level  of  perception  of  community  risk 
(table  1.32).  From  the  54  comparisons  made,  3 significant  differences  would  be  expected  by  chance.  Six  of 
the  significant  results  showed  beneficial  marginal  impacts  for  high-risk  students:  increases  in  Positive  School 
Orientation  (Year  2)  and  declines  in  Problem  Behavior  (Teacher  Report  in  Year  2),  ADHD-related  behavior 
(Year  2),  and  Victimization  at  School  (all  3 years).  Only  1 of  these  6 was  supported  by  the  subgroup -specific 
regressions  with  the  finding  of  a beneficial  impact  of  the  decline  in  Victimization  at  School  (Year  2)  for  high- 
risk  students. 

The  4 detrimental  marginal  impacts  for  high-risk  students  included  declines  in  Empathy  (Year  3)  and 
Altruistic  Behavior  (Primary  Caregiver  Report  in  Year  3),  and  increases  in  Problem  Behavior  (Primary 
Caregiver  Report  in  Years  1 and  2).  In  all  four  cases,  the  subgroup-specific  regressions  supported  the  finding 
of  detrimental  impact  on  all  4 outcomes  for  high-risk  students. 
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Table  1.32.  Combined-program  impacts  on  child  outcomes,  by  initial  perceptions  of  community  risk 


Scale-Report 

Social  and  Emotional 
Competence  Domain 


Year  1 

(Spring  3rd  grade) 

Effect  size  at  risk  level 

(p-value) 

Aver- 


Marg- 

inal 

effect4 

r-value) 


Year  2 

(Spring  4th  grade) 

Effect  size  at  risk  level 

(p-value) 

Aver- 


Marg- 

inal 

effect4 

o-value) 


Year  3 

(Spring  5th  grade) 

Effect  size  at  risk  level 

(p-value) 

Aver- 


Self-Efficacy  for  Peer 

-0.025 

-0.029 

-0.032 

-0.005 

-0.121 A 

-0.064 

-0.008 

0.085 

-0.061 

-0.024 

0.014 

0.056 

Interactions-CR  (+) 

(0.642) 

(0.491) 

(0.550) 

(0.923) 

(0.067) 

(0.204) 

(0.905) 

(0.159) 

(0.323) 

(0.596) 

(0.832) 

(0.395) 

Normative  Beliefs  About 

0.033 

-0.003 

-0.040 

-0.055 

0.005 

-0.008 

-0.021 

0.019 

-0.035 

-0.012 

0.011 

0.035 

Aggression-CR  (-) 

(0.582) 

(0.948) 

(0.504) 

(0.328) 

(0.959) 

(0.914) 

(0.807) 

(0.768) 

(0.699) 

(0.873) 

(0.901) 

(0.632) 

Empathy-CR  (+) 

0.087 

0.073 

0.059 

-0.021 

0.013 

-0.005 

-0.022 

-0.026 

0.080 

-0.054 

-0.187* 

-0.200  ** 

(0.171) 

(0.153) 

(0.337) 

(0.698) 

(0.860) 

(0.934) 

(0.747) 

(0.671) 

(0.341) 

(0.423) 

(0.023) 

(0.006) 

Behavior  Domain 

Altruistic  Behavior- 

-0.035 

-0.062 

-0.089A 

-0.041 

-0.060 

-0.031 

-0.001 

0.045 

-0.017 

-0.023 

-0.030 

-0.010 

CR  (+) 

(0.523) 

(0.136) 

(0.095) 

(0.422) 

(0.356) 

(0.539) 

(0.990) 

(0.460) 

(0.808) 

(0.647) 

(0.661) 

(0.881) 

Altruistic  Behavior- 

0.030 

0.036 

0.042 

0.008 

0.079 

0.004 

-0.070 

-0.1 1 2A 

0.1 29A 

-0.024 

-0.176* 

-0.228** 

PCR  (+) 

(0.538) 

(0.315) 

(0.386) 

(0.864) 

(0.205) 

(0.927) 

(0.238) 

(0.060) 

(0.082) 

(0.670) 

(0.015) 

(0.001) 

Altruistic  Behavior- 

0.159 

0.096 

0.033 

-0.094 

-0.009 

0.027 

0.063 

0.054 

-0.004 

-0.013 

-0.023 

-0.014 

TRS  (+) 

(0.141) 

(0.333) 

(0.753) 

(0.104) 

(0.961) 

(0.875) 

(0.723) 

(0.451) 

(0.978) 

(0.919) 

(0.871) 

(0.853) 

Positive  Social 

0.001 

-0.014 

-0.030 

-0.023 

0.044 

0.016 

-0.012 

-0.042 

0.088 

0.013 

-0.063 

-0.1 12A 

Behavior-PCR  (+) 

(0.977) 

(0.657) 

(0.476) 

(0.576) 

(0.407) 

(0.693) 

(0.805) 

(0.392) 

(0.220) 

(0.822) 

(0.359) 

(0.076) 

Positive  Social 

0.002 

-0.004 

-0.011 

-0.010 

-0.074 

-0.045 

-0.015 

0.044 

0.096 

0.038 

-0.020 

-0.087 

Behavior-TRS  (+) 

(0.966) 

(0.926) 

(0.834) 

(0.800) 

(0.405) 

(0.569) 

(0.859) 

(0.441) 

(0.295) 

(0.631) 

(0.819) 

(0.187) 

Problem  Behavior- 

0.022 

-0.008 

-0.037 

-0.044 

-0.022 

-0.027 

-0.032 

-0.008 

-0.002 

0.032 

0.067 

0.052 

CR  (-) 

(0.732) 

(0.880) 

(0.543) 

(0.391) 

(0.798) 

(0.715) 

(0.698) 

(0.900) 

(0.982) 

(0.680) 

(0.457) 

(0.457) 

Problem  Behavior- 

-0.100A 

0.005 

0.109* 

0.157*** 

-0.045 

0.028 

0.101* 

0.109* 

-0.005 

0.000 

0.006 

0.009 

PCR  (-) 

(0.053) 

(0.909) 

(0.028) 

(0.001) 

(0.381) 

(0.446) 

(0.041) 

(0.032) 

(0.940) 

(0.994) 

(0.929) 

(0.896) 

Problem  Behavior- 

0.044 

0.054 

0.063 

0.014 

0.087 

-0.001 

-0.089 

-0.132* 

-0.087 

-0.047 

-0.007 

0.059 

TRS  (-) 

(0.464) 

(0.304) 

(0.278) 

(0.732) 

(0.268) 

(0.994) 

(0.243) 

(0.022) 

(0.370) 

(0.577) 

(0.939) 

(0.374) 

ADHD-Related  Behavior- 

-0.016 

0.022 

0.028 

0.008 

0.062 

-0.021 

-0.104 

-0.124* 

-0.073 

-0.102 

-0.132 

-0.045 

TRS  (-) 

(0.748) 

(0.605) 

(0.576) 

(0.835) 

(0.389) 

(0.722) 

(0.132) 

(0.029) 

(0.375) 

(0.134) 

(0.100) 

(0.494) 

See  notes  at  end  of  table. 
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Table  1.32.  Combined-program  impacts  on  child  outcomes,  by  initial  perceptions  of  community  risk — Continued 

Year  1 Year  2 Year  3 

(Spring  3rd  grade) (Spring  4th  grade) (Spring  5th  grade) 


Effect  size  at  risk  level 
(p-value) 

Marg- 

inal 

Effect  size  at  risk  level 
(p-value) 

Marg- 

inal 

Effect  size  at  risk  level 
(p-value) 

Marg- 

inal 

Scale-Report 

Low1 

Aver- 

age2 

High3 

effect4 

(p-value) 

Low1 

Aver- 

age2 

High3 

effect4 

(p-value) 

Low1 

Aver- 

age2 

High3 

effect4 

(p-value) 

Academics  Domain 

Engagement  with 

-0.016 

-0.054 

-0.091 A 

-0.056 

-0.020 

-0.003 

0.014 

0.026 

-0.052 

-0.053 

-0.055 

-0.002 

Learning-CR  (+) 

(0.766) 

(0.187) 

(0.095) 

(0.302) 

(0.728) 

(0.943) 

(0.804) 

(0.657) 

(0.463) 

(0.313) 

(0.439) 

(0.972) 

Academic  Competence 

0.017 

-0.002 

-0.014 

-0.023 

0.036 

-0.015 

-0.066 

-0.077A 

0.058 

0.017 

-0.024 

-0.061 

and  Motivation-TRS  (+) 

(0.640) 

(0.956) 

(0.695) 

(0.489) 

(0.483) 

(0.719) 

(0.188) 

(0.092) 

(0.396) 

(0.762) 

(0.720) 

(0.275) 

Perceptions  of  School 
Climate  Domain 

Positive  School 

0.011 

0.013 

0.015 

-0.003 

-0.086 

0.030 

0.146 

0.174** 

-0.091 

-0.094 

-0.098 

-0.005 

Orientation-CR  (+) 

(0.895) 

(0.855) 

(0.850) 

(0.957) 

(0.361) 

(0.719) 

(0.111) 

(0.007) 

(0.378) 

(0.293) 

(0.331) 

(0.945) 

Negative  School 

-0.047 

-0.040 

-0.034 

0.010 

-0.079 

-0.055 

-0.031 

0.036 

-0.063 

-0.052 

-0.041 

0.017 

Orientation-CR  (-) 

(0.435) 

(0.395) 

(0.561) 

(0.855) 

(0.342) 

(0.432) 

(0.700) 

(0.561) 

(0.435) 

(0.429) 

(0.607) 

(0.805) 

Student  Afraid  at 

-0.062 

-0.061 

-0.060 

-0.002 

-0.017 

-0.078 

-0.1 39A 

-0.091 

0.003 

-0.010 

-0.023 

-0.020 

School-CR  (-) 

(0.331) 

(0.233) 

(0.334) 

(0.976) 

(0.836) 

(0.240) 

(0.073) 

(0.157) 

(0.973) 

(0.881) 

(0.777) 

(0.783) 

Victimization  at 

0.094A 

-0.001 

-0.093A 

-0.139** 

0.068 

-0.035 

-0.138* 

-0.154* 

0.132' 

v 0.021 

-0.090 

-0.166* 

School-CR  (-) 

(0.093) 

(0.988) 

(0.091) 

(0.008) 

(0.298) 

(0.485) 

(0.030) 

(0.011) 

(0.066) 

(0.702) 

(0.205) 

(0.016) 

* Significantly  different  from  zero  at  the  .05  level. 

**  Significantly  different  from  zero  at  the  .01  level. 

***  Significantly  different  from  zero  at  the  .001  level. 

A Significantly  different  from  zero  at  the  .10  to  > .05  level. 

1 One  standard  deviation  below  the  mean  risk  level. 

2 At  the  mean  risk  level. 

3 One  standard  deviation  above  the  mean  risk  level. 

4 Change  in  impact  as  risk  level  increases  by  one  unit. 

NOTE:  Abbreviations  are 

CR:  Child  Report 

PCR:  Primary  Caregiver  Report 

TRS:  Teacher  Report  on  Student 

ADHD:  Attention  deficit  hyperactivity  disorder 

The  +/-  signs  in  parentheses  indicate  the  direction  of  a beneficial  outcome.  All  impact  estimates  were  calculated  using  regression  models  in  which  each  program  and  school  within  a 
program  was  weighted  equally.  The  standard  errors  of  all  estimates  account  for  design  effects  due  to  unequal  weighting  and  the  clustering  of  students  within  schools.  The  effect  size 
was  calculated  by  dividing  the  estimated  impact  by  the  standard  deviation  of  the  outcome  measure  for  the  control  group. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Child  Behavior  Risk  (Teacher  and  Primary  Caregiver  Reports) 

Over  the  3 years,  the  Teacher  Report  showed  10  significant  marginal  effects  regarding  level  of  child  behavior 
risk  (table  1.33).  From  the  54  comparisons  made,  3 significant  differences  would  be  expected  by  chance. 
Seven  of  the  significant  results  showed  beneficial  marginal  impacts  for  high-risk  students:  increases  in 
Altruistic  Behavior  (Primary  Caregiver  Report  in  Year  1)  and  Academic  Competence  (Years  2 and  3),  and 
declines  in  Normative  Beliefs  About  Aggression  (Year  2),  Problem  Behavior  (Child  Report  and  Teacher 
Report  in  Year  2),  and  Negative  School  Orientation  (Year  2).  Results  from  the  subgroup -specific  regressions 
supported  the  beneficial  impacts  for  higher  risk  students  for  Altruistic  Behavior  and  Negative  School 
Orientation,  and  also  provided  evidence  that  the  marginal  effect  on  Academic  Competence  (Year  2)  was  due 
to  a detrimental  impact  on  low-risk  students. 

The  3 detrimental  marginal  impacts  for  high-risk  students  included  declines  in  Engagement  with  Learning 
(Year  1)  and  increases  in  Problem  Behavior  (Primary  Caregiver  Report  in  Year  3)  and  Student  Afraid  at 
School  (Year  1).  Results  from  the  subgroup-specific  regressions  supported  the  interpretation  that  there  were 
detrimental  impacts  on  Engagement  with  Learning  for  high-risk  students  but  that  the  marginal  impacts  on 
Problem  Behavior  and  Student  Afraid  at  School  were  due  to  beneficial  impacts  on  low-risk  students. 

When  child  behavior  risk  was  measured  using  the  Primary  Caregiver  Report,  there  were  10  significant 
marginal  effects  in  contrast  to  the  3 expected  by  chance  from  the  54  comparisons  (table  1.34).  Eight  of  these 
showed  beneficial  marginal  impacts  for  high-risk  students:  increases  in  Empathy  (Years  1 and  2),  Altruistic 
Behavior  (Teacher  Report  in  Year  2),  Positive  Social  Behavior  (Primary  Caregiver  Report  in  Year  2),  and 
Engagement  with  Learning  (Year  2),  and  declines  in  Normative  Beliefs  About  Aggression  (Year  2)  and 
Negative  School  Orientation  (Years  1 and  2).  Results  from  the  subgroup-specific  regressions  supported  the 
interpretation  that  there  were  beneficial  impacts  on  Empathy  (Year  1)  and  Negative  School  Orientation  (Year 
2)  for  high-risk  students.  The  2 detrimental  marginal  impacts  for  high-risk  students  were  found  for  Problem 
Behavior  (Primary  Caregiver  Report  in  Years  1 and  3). 
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Table  1.33.  Combined-program  impacts  on  child  outcomes,  by  initial  child  behavior  risk  as  reported  by  teacher 

Year  1 Year  2 Year  3 


(Spring  3rd  grade) (Spring  4th  grade) (Spring  5th  grade) 


Effect  size  at  risk  level 
(p-value) 

Marg- 

inal 

Effect  size  at  risk  level 
(p-value) 

Marg- 

inal 

Effect  size  at  risk  level 
(p-value) 

Marg- 

inal 

Scale-Report 

Low1 

Aver- 

age2 

High3 

effect4 

(p-value) 

Low1 

Aver- 

age2 

High3 

effect4 

(p-value) 

Low1 

Aver- 

age2 

High3 

effect4 

(p-value) 

Social  and  Emotional 
Competence  Domain 

Self-Efficacy  for  Peer 

0.006 

-0.037 

-0.080 

-0.004 

-0.040 

-0.059 

-0.077 

-0.002 

-0.028 

-0.041 

-0.053 

-0.001 

Interactions-CR  (+) 

(0.904) 

(0.336) 

(0.109) 

(0.169) 

(0.510) 

(0.230) 

(0.228) 

(0.640) 

(0.653) 

(0.397) 

(0.424) 

(0.771) 

Normative  Beliefs  About 

-0.010 

-0.015 

-0.019 

0.000 

0.127 

0.007 

-0.113 

-0.012** 

0.014 

0.021 

0.028 

0.001 

Aggression-CR  (-) 

(0.851) 

(0.742) 

(0.734) 

(0.897) 

(0.134) 

(0.923) 

(0.191) 

(0.005) 

(0.867) 

(0.770) 

(0.741) 

(0.870) 

Empathy-CR  (+) 

0.035 

0.044 

0.052 

0.001 

-0.093 

-0.018 

0.057 

0.007A 

-0.027 

-0.061 

-0.095 

-0.003 

(0.575) 

(0.418) 

(0.404) 

(0.786) 

(0.171) 

(0.748) 

(0.416) 

(0.055) 

(0.728) 

(0.351) 

(0.234) 

(0.430) 

Behavior  Domain 

Altruistic  Behavior- 

-0.071 

-0.077* 

-0.083A 

-0.001 

-0.074 

-0.048 

-0.022 

0.003 

-0.036 

-0.029 

-0.023 

0.001 

CR  (+) 

(0.136) 

(0.036) 

(0.085) 

(0.844) 

(0.227) 

(0.321) 

(0.734) 

(0.510) 

(0.578) 

(0.561) 

(0.740) 

(0.878) 

Altruistic  Behavior- 

-0.065 

0.046 

0.156** 

0.011** 

-0.011 

-0.025 

-0.040 

-0.001 

0.001 

-0.063 

-0.128 

-0.006 

PCR  (+) 

(0.217) 

(0.265) 

(0.003) 

(0.001) 

(0.855) 

(0.565) 

(0.505) 

(0.706) 

(0.987) 

(0.308) 

(0.119) 

(0.187) 

Altruistic  Behavior- 

0.083 

0.119 

0.156 

0.004 

0.013 

0.030 

0.048 

0.002 

-0.024 

-0.014 

-0.004 

0.001 

TRS  (+) 

(0.531) 

(0.353) 

(0.237) 

(0.222) 

(0.941) 

(0.858) 

(0.783) 

(0.673) 

(0.853) 

(0.910) 

(0.975) 

(0.809) 

Positive  Social 

-0.014 

-0.016 

-0.017 

0.000 

-0.028 

0.021 

0.069 

0.005 

-0.008 

0.021 

0.050 

0.003 

Behavior-PCR  (+) 

(0.716) 

(0.600) 

(0.667) 

(0.961) 

(0.588) 

(0.601) 

(0.192) 

(0.148) 

(0.898) 

(0.700) 

(0.477) 

(0.481) 

Positive  Social 

-0.049 

-0.017 

0.014 

0.003 

-0.067 

-0.056 

-0.045 

0.001 

0.037 

0.038 

0.038 

0.000 

Behavior-TRS  (+) 

(0.355) 

(0.717) 

(0.789) 

(0.145) 

(0.433) 

(0.477) 

(0.599) 

(0.745) 

(0.663) 

(0.630) 

(0.663) 

(0.993) 

Problem  Behavior- 

-0.012 

0.036 

0.061 

0.002 

0.075 

-0.007 

-0.089 

-0.008* 

0.042 

0.023 

0.003 

-0.002 

CR  (-) 

(0.840) 

(0.480) 

(0.312) 

(0.424) 

(0.348) 

(0.922) 

(0.275) 

(0.032) 

(0.621) 

(0.765) 

(0.969) 

(0.630) 

Problem  Behavior- 

-0.019 

-0.005 

0.009 

0.001 

0.017 

-0.007 

-0.031 

-0.002 

-0.202**  -0.051 

0.101 

0.015*** 

PCR  (-) 

(0.698) 

(0.898) 

(0.857) 

(0.619) 

(0.735) 

(0.849) 

(0.549) 

(0.484) 

(0.009) 

(0.423) 

(0.211) 

(0.001) 

Problem  Behavior- 

-0.007 

0.048 

0.088A 

0.004A 

0.064 

-0.011 

-0.085 

-0.007* 

-0.014 

-0.030 

-0.046 

-0.002 

TRS  (-) 

(0.890) 

(0.322) 

(0.099) 

(0.078) 

(0.389) 

(0.872) 

(0.255) 

(0.029) 

(0.873) 

(0.697) 

(0.593) 

(0.670) 

ADHD-Related  Behavior- 

-0.031 

0.039 

0.048 

0.001 

0.036 

-0.007 

-0.051 

-0.004 

-0.080 

-0.071 

-0.062 

0.001 

TRS  (-) 

(0.512) 

(0.335) 

(0.299) 

(0.689) 

(0.589) 

(0.902) 

(0.456) 

(0.199) 

(0.298) 

(0.296) 

(0.431) 

(0.808) 

See  notes  at  end  of  table. 
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Table  1.33.  Combined-program  impacts  on  child  outcomes,  by  initial  child  behavior  risk  as  reported  by  teacher — Continued 


Year  1 

Year  2 

Year  3 

(Spring  3rd  grade) 

(Spring  4th  grade) 

(Spring  5th  grade) 

Effect  size  at  risk  level 
(p-value) 

Marg- 

inal 

Effect  size  at  risk  level 
(p-value) 

Marg- 

inal 

Effect  size  at  risk  level  Marg- 

(p-value)  inal 

Aver- 

CD 

CD 

O 

-p-' 

Aver- 

effect4 

Aver-  effectd 

Scale-Report 

Low1  age2  High3 

(p-value) 

Low1  age2  High3 

(p-value) 

Low1  age2  High3  (p-value) 

Academics  Domain 

Engagement  with 

0.041 

-0.051 

-0.143** *** 

-0.009** 

-0.069 

-0.016 

0.038 

0.005 

0.006 

-0.043 

-0.092 

-0.005 

Learning-CR  (+) 

(0.413) 

(0.185) 

(0.006) 

(0.007) 

(0.228) 

(0.711) 

(0.532) 

(0.187) 

(0.925) 

(0.420) 

(0.199) 

(0.272) 

Academic  Competence 

-0.052 

-0.022 

0.008 

0.003 

-0.108* 

-0.032 

0.044 

0.007** 

-0.088 

-0.006 

0.076 

0.008* 

and  Motivation-TRS  (+) 

(0.123) 

(0.429) 

(0.808) 

(0.111) 

(0.035) 

(0.455) 

(0.391) 

(0.008) 

(0.149) 

(0.909) 

(0.225) 

(0.013) 

Perceptions  of  School 
Climate  Domain 
Positive  School 

0.013 

-0.012 

-0.037 

-0.002 

0.040 

0.043 

0.046 

0.000 

-0.069 

-0.065 

-0.062 

0.000 

Orientation-CR  (+) 

(0.866) 

(0.867) 

(0.635) 

(0.430) 

(0.668) 

(0.612) 

(0.622) 

(0.933) 

(0.472) 

(0.452) 

(0.525) 

(0.935) 

Negative  School 

-0.019 

-0.012 

-0.006 

-0.001 

0.057 

-0.072 

-0.202* 

-0.013** 

0.027 

-0.048 

-0.122 

-0.007A 

Orientation-CR  (-) 

(0.749) 

(0.801) 

(0.915) 

(0.843) 

(0.467) 

(0.300) 

(0.013) 

(0.001) 

(0.728) 

(0.478) 

(0.128) 

(0.072) 

Student  Afraid  at 

-0.140* 

-0.068 

0.004 

0.007* 

-0.041 

-0.054 

-0.066 

-0.001 

0.030 

0.033 

0.035 

0.000 

School-CR  (-) 

(0.018) 

(0.162) 

(0.942) 

(0.029) 

(0.572) 

(0.384) 

(0.375) 

(0.755) 

(0.705) 

(0.631) 

(0.664) 

(0.949) 

Victimization  at 

-0.008 

-0.005 

-0.019 

-0.001 

-0.010 

-0.043 

-0.076 

-0.003 

0.045 

0.011 

-0.023 

-0.003 

School-CR  (-) 

(0.873) 

(0.896) 

(0.716) 

(0.670) 

(0.867) 

(0.337) 

(0.208) 

(0.390) 

(0.511) 

(0.841) 

(0.751) 

(0.425) 

* Significantly  different  from  zero  at  the  .05  level. 


**  Significantly  different  from  zero  at  the  .01  level. 

***  Significantly  different  from  zero  at  the  .001  level. 

A Significantly  different  from  zero  at  the  .10  to  > .05  level. 

1 One  standard  deviation  below  the  mean  risk  level. 

2 At  the  mean  risk  level. 

3 One  standard  deviation  above  the  mean  risk  level. 

4 Change  in  impact  as  risk  level  increases  by  one  unit. 

NOTE:  Abbreviations  are 

CR:  Child  Report 

PCR:  Primary  Caregiver  Report 

TRS:  Teacher  Report  on  Student 

ADHD:  Attention  deficit  hyperactivity  disorder 

The  +/-  signs  in  parentheses  indicate  the  direction  of  a beneficial  outcome.  All  impact  estimates  were  calculated  using  regression  models  in  which  each  program  and  school  within  a 
program  was  weighted  equally.  The  standard  errors  of  all  estimates  account  for  design  effects  due  to  unequal  weighting  and  the  clustering  of  students  within  schools.  The  effect  size 
was  calculated  by  dividing  the  estimated  impact  by  the  standard  deviation  of  the  outcome  measure  for  the  control  group. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Table  1.34.  Combined-program  impacts  on  child  outcomes,  by  child  behavior  risk  as  reported  by  primary  caregiver 


Scale-Report 

Social  and  Emotional 
Competence  Domain 
Self-Efficacy  for  Peer 
Interactions-CR  (+) 

Normative  Beliefs 
About  Aggression-CR  (-) 

Empathy-CR  (+) 


Behavior  Domain 


Altruistic  Behavior- 

-0.110* 

-0.064 

-0.017 

0.005 

-0.090 

-0.031 

0.029 

0.006 

-0.074 

-0.030 

0.014 

0.004 

CR  (+) 

(0.029) 

(0.108) 

(0.734) 

(0.128) 

(0.148) 

(0.539) 

(0.651) 

(0.118) 

(0.248) 

(0.547) 

(0.838) 

(0.287) 

Altruistic  Behavior- 

0.081 

0.037 

-0.007 

-0.004 

-0.040 

0.001 

0.042 

0.004 

-0.030 

-0.038 

-0.046 

-0.001 

PCR  (+) 

(0.199) 

(0.350) 

(0.890) 

(0.178) 

(0.500) 

(0.986) 

(0.485) 

(0.281) 

(0.695) 

(0.540) 

(0.548) 

(0.856) 

Altruistic  Behavior- 

0.077 

0.072 

0.066 

-0.001 

-0.049 

0.029 

0.107 

0.008* 

-0.010 

-0.015 

-0.019 

0.000 

TRS  (+) 

(0.431) 

(0.439) 

(0.496) 

(0.860) 

(0.780) 

(0.864) 

(0.542) 

(0.050) 

(0.938) 

(0.907) 

(0.885) 

(0.910) 

Positive  Social 

0.013 

-0.027 

-0.066A 

-0.004 

-0.086A 

-0.007 

0.072 

0.008* 

-0.013 

-0.003 

0.008 

0.001 

Behavior-PCR  (+) 

(0.745) 

(0.384) 

(0.097) 

(0.127) 

(0.083) 

(0.852) 

(0.147) 

(0.015) 

(0.845) 

(0.961) 

(0.909) 

(0.786) 

Positive  Social 

-0.010 

-0.009 

-0.009 

0.000 

-0.016 

0.038 

-0.061 

-0.002 

0.010 

0.033 

0.056 

0.002 

Behavior-TRS  (+) 

(0.849) 

(0.835) 

(0.854) 

(0.992) 

(0.854) 

(0.630) 

(0.480) 

(0.483) 

(0.904) 

(0.674) 

(0.522) 

(0.528) 

Problem  Behavior- 

0.025 

-0.003 

-0.031 

-0.003 

0.014 

-0.037 

-0.089 

-0.005 

0.040 

0.032 

0.024 

-0.001 

CR  (-) 

(0.677) 

(0.953) 

(0.603) 

(0.346) 

(0.865) 

(0.612) 

(0.281) 

(0.143) 

(0.638) 

(0.677) 

(0.781) 

(0.830) 

Problem  Behavior- 

-0.076 

0.003 

0.081 A 

0.008** 

-0.021 

0.036 

0.092A 

0.006 

-0.101 

-0.017 

0.066 

0.008* 

PCR  (-) 

(0.109) 

(0.943) 

(0.083) 

(0.006) 

(0.691) 

(0.363) 

(0.080) 

(0.101) 

(0.135) 

(0.749) 

(0.328) 

(0.043) 

Problem  Behavior- 

0.043 

0.053 

0.062 

0.001 

-0.010 

-0.008 

-0.006 

0.000 

-0.017 

-0.044 

-0.071 

-0.003 

TRS  (-) 

(0.428) 

(0.288) 

(0.256) 

(0.686) 

(0.894) 

(0.908) 

(0.940) 

(0.949) 

(0.846) 

(0.587) 

(0.427) 

(0.457) 

ADHD-Related 

0.000 

0.020 

0.041 

0.002 

-0.002 

-0.021 

-0.040 

-0.002 

-0.107 

-0.096 

-0.085 

0.001 

Behavior-TRS  (-) 

(0.996) 

(0.619) 

(0.379) 

(0.357) 

(0.982) 

(0.732) 

(0.563) 

(0.556) 

(0.168) 

(0.163) 

(0.278) 

(0.763) 

See  notes  to  at  end  of  table. 


Year  1 Year  2 Year  3 

(Spring  3rd  grade) (Spring  4th  grade) (Spring  5th  grade) 


Effect  size  at  risk  level 
(p-value) 

Marg- 

Effect  size  at  risk  level 
(p-value) 

Marg- 

Effect  size  at  risk  level 
(p-value) 

Marg- 

Aver- 

effect4 

Aver- 

effect4 

Aver- 

effect4 

Low1  age2  High3 

(p-value) 

Low1  age2  High3 

(p-value) 

Low1  age2  High3 

(p-value) 

0.027 

-0.031 

-0.090A 

-0.006A 

-0.015 

-0.054 

-0.093 

-0.004 

-0.001 

-0.022 

-0.043 

-0.002 

(0.582) 

(0.421) 

(0.072) 

(0.061) 

(0.798) 

(0.247) 

(0.125) 

(0.295) 

(0.986) 

(0.609) 

(0.474) 

(0.599) 

0.020 

-0.012 

-0.044 

-0.003 

0.071 

-0.010 

-0.092 

-0.008* 

-0.017 

-0.010 

-0.004 

0.001 

(0.725) 

(0.802) 

(0.446) 

(0.338) 

(0.403) 

(0.893) 

(0.287) 

(0.036) 

(0.847) 

(0.893) 

(0.965) 

(0.876) 

-0.013 

0.077 

0.166** 

0.009** 

-0.083 

-0.007 

0.068 

0.008* 

-0.021 

-0.063 

-0.105 

-0.004 

(0.838) 

(0.148) 

(0.008) 

(0.005) 

(0.217) 

(0.897) 

(0.318) 

(0.042) 

(0.786) 

(0.352) 

(0.189) 

(0.306) 
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Table  1.34.  Combined-program  impacts  on  child  outcomes,  by  child  behavior  risk  as  reported  by  primary  caregiver — Continued 


Year  1 

Year  2 

Year  3 

(Spring  3rd  grade) 

(Spring  4th  grade) 

(Spring  5th  grade) 

Effect  size  at  risk  level 

Marg- 

Effect size  at  risk  level 

Marg- 

Effect size  at  risk  level 

Marg- 

(p-value) 

inal 

(p-value) 

inal 

(p-value) 

inal 

Scale-Report 

Low1 

Aver- 

age2 

High3 

effect4 

(p-value) 

Low1 

Aver- 

age2 

High3 

effect4 

(p-value) 

Low1 

Aver- 

age2 

High3 

effect4 

(p-value) 

Academics  Domain 

Engagement  with 

-0.074 

-0.057 

-0.039 

0.002 

-0.089 

-0.004 

0.081 

0.009* 

-0.004 

-0.057 

-0.110 

-0.005 

Learning-CR  (+) 

(0.151) 

(0.149) 

(0.443) 

(0.599) 

(0.127) 

(0.925) 

(0.178) 

(0.028) 

(0.954) 

(0.278) 

(0.109) 

(0.210) 

Academic  Competence 

-0.001 

-0.001 

-0.003 

0.000 

-0.044 

-0.013 

0.019 

0.003 

-0.025 

0.005 

0.034 

0.003 

and  Motivation-TRS  (+) 

(0.979) 

(0.970) 

(0.930) 

(0.923) 

(0.365) 

(0.754) 

(0.705) 

(0.247) 

(0.702) 

(0.932) 

(0.600) 

(0.350) 

Perceptions  of  School 
Climate  Domain 

Positive  School 

-0.050 

0.008 

0.066 

0.006A 

-0.009 

0.053 

0.115 

0.006A 

-0.022 

-0.080 

-0.138 

-0.006 

Orientation-CR  (+) 

(0.534) 

(0.912) 

(0.408) 

(0.067) 

(0.918) 

(0.532) 

(0.215) 

(0.092) 

(0.819) 

(0.367) 

(0.158) 

(0.143) 

Negative  School 

0.049 

-0.034 

-0.1 1 6A 

-0.008** 

0.055 

0.055 

-0.178* 

-0.012** 

-0.017 

-0.054 

-0.090 

-0.004 

Orientation-CR  (-) 

(0.418) 

(0.513) 

(0.055) 

(0.009) 

(0.492) 

(0.492) 

(0.030) 

(0.002) 

(0.829) 

(0.433) 

(0.254) 

(0.341) 

Student  Afraid  at 

-0.070 

-0.055 

-0.041 

0.001 

-0.073 

-0.068 

-0.062 

0.001 

0.068 

0.006 

-0.056 

-0.006 

School-CR  (-) 

(0.247) 

(0.274) 

(0.494) 

(0.653) 

(0.317) 

(0.285) 

(0.406) 

(0.879) 

(0.385) 

(0.932) 

(0.478) 

(0.123) 

Victimization  at 

0.026 

-0.005 

-0.016 

-0.002 

-0.032 

-0.040 

-0.048 

-0.001 

-0.010 

0.010 

0.030 

0.002 

School-CR  (-) 

(0.621) 

(0.903) 

(0.763) 

(0.508) 

(0.598) 

(0.407) 

(0.432) 

(0.822) 

(0.888) 

(0.865) 

(0.678) 

(0.621) 

* Significantly  different  from  zero  at  the  .05  level. 


**  Significantly  different  from  zero  at  the  .01  level. 

A Significantly  different  from  zero  at  the  .10  to  > .05  level. 

1 One  standard  deviation  below  the  mean  risk  level. 

2 At  the  mean  risk  level. 

3 One  standard  deviation  above  the  mean  risk  level. 

4 Change  in  impact  as  risk  level  increases  by  one  unit. 

NOTE:  Abbreviations  are 

CR:  Child  Report 

PCR:  Primary  Caregiver  Report 

TRS:  Teacher  Report  on  Student 

ADHD:  Attention  deficit  hyperactivity  disorder 

The  +/-  signs  in  parentheses  indicate  the  direction  of  a beneficial  outcome.  All  impact  estimates  were  calculated  using  regression  models  in  which  each  program  and  school  within  a 
program  was  weighted  equally.  The  standard  errors  of  all  estimates  account  for  design  effects  due  to  unequal  weighting  and  the  clustering  of  students  within  schools.  The  effect  size 
was  calculated  by  dividing  the  estimated  impact  by  the  standard  deviation  of  the  outcome  measure  for  the  control  group. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Fidelity  of  Implementation 

There  were  5 cases  of  significant  differences  between  the  associations  of  the  SACD  programs  with  the 
outcomes  for  schools  with  different  levels  of  fidelity  of  implementation  (table  1.35).  From  the  54 
comparisons  made,  3 significant  differences  would  be  expected  by  chance.  When  examining  the  results  of  the 
/ tests  from  subgroup-specific  regressions  for  these  5 cases,  4 cases  were  based  on  significant  detrimental 
program  associations  for  low-fidelity  schools:  declines  in  Altruistic  Behavior  (Child  Report  in  Year  1)  and 
Positive  School  Orientation  (in  Year  3)  and  increases  in  Problem  Behavior  (Child  Report  in  Year  3)  and 
Negative  School  Orientation  (in  Year  3).  One  was  linked  to  a beneficial  (though  not  significant)  association 
for  high-fidelity  schools  (a  decline  in  Student  Afraid  at  School  in  Year  3). 

As  a sensitivity  test  for  the  Year  2 and  Year  3 data,  the  schools  were  divided  into  only  two  categories  (high  or 
low  implementers)  and  the  Year  1-type  analysis  was  done  (using  two  fidelity  subgroups  based  on  the  most 
recent  year  of  fidelity  data;  these  data  are  not  shown  in  a table).  No  significant  differences  in  associations 
were  found  between  high  and  low  implementers  in  this  analysis.  Because  the  few  significant  findings  from  the 
first  analysis  were  primarily  due  to  detrimental  associations  between  low  fidelity  and  outcomes  (rather  than 
beneficial  associations  between  high  fidelity  and  outcomes),  and  because  there  were  no  significant  findings  in 
the  sensitivity  analysis,  the  fidelity  analyses  provide  little  support  for  the  hypothesis  of  a relationship  between 
high  fidelity  and  more  beneficial  outcomes. 
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Table  1.35.  Combined-program  results  for  child  outcomes,  by  fidelity  of  implementation 


Year  1 Year  2 Year  3 

(Spring  3rd  grade)  (Spring  4th  grade) (Spring  5th  grade) 


Scale-Report 

Effect  size 
(p-value) 

p-value 

for 

Effect  size 
(p-value) 

p-value 

for 

Effect  size 
(p-value) 

p-value 

for 

Low 

High 

differ- 

ence1 

Low 

Mixed 

High 

differ- 

ence1 

Low 

Mixed 

High 

differ- 

ence1 

Social  and  Emotional 

Competence  Domain 

Self-Efficacy  for  Peer 

-0.02 

-0.04 

0.882 

-0.1 2A 

0.02 

-0.1 3A 

0.730 

-0.09 

-0.01 

-0.07 

0.841 

Interactions-CR  (+) 

(0.758) 

(0.397) 

(0.087) 

(0.827) 

(0.063) 

(0.364) 

(0.909) 

(0.291) 

Normative  Beliefs  About 

0.04 

-0.06 

0.160 

0.06 

0.04 

-0.14 

0.068A 

0.1 7A 

-0.04 

-0.10 

0.069A 

Aggression-CR  (-) 

(0.438) 

(0.300) 

(0.419) 

(0.584) 

(0.148) 

(0.091) 

(0.640) 

(0.235) 

Empathy-CR  (+) 

-0.01 

0.15* 

0.074A 

-0.08 

-0.03 

0.02 

0.531 

-0.15 

-0.08 

0.05 

0.298 

(0.851) 

(0.024) 

(0.242) 

(0.736) 

(0.797) 

(0.112) 

(0.359) 

(0.449) 

Behavior  Domain 

Altruistic  Behavior- 

-0.13** 

0.01 

0.033* 

-0.05 

-0.08 

-0.08 

0.907 

-0.05 

-0.03 

-0.03 

0.904 

OR  (+) 

(0.010) 

(0.810) 

(0.474) 

(0.332) 

(0.336) 

(0.508) 

(0.663) 

(0.589) 

Altruistic  Behavior- 

0.10A 

0.04 

0.283 

-0.03 

0.13 

-0.02 

0.248 

-0.05 

0.00 

-0.05 

0.896 

PCR  (+) 

(0.061) 

(0.407) 

(0.693) 

(0.135) 

(0.660) 

(0.565) 

(0.995) 

(0.482) 

Altruistic  Behavior- 

0.10 

0.13 

0.888 

-0.10 

0.07 

0.12 

0.832 

0.25 

-0.12 

-0.21 

0.324 

TRS  (+) 

(0.435) 

(0.322) 

(0.589) 

(0.845) 

(0.648) 

(0.359) 

(0.358) 

(0.366) 

Positive  Social 

0.00 

-0.02 

0.692 

0.05 

0.11 

-0.01 

0.692 

0.01 

0.08 

-0.02 

0.719 

Behavior-PCR  (+) 

(0.958) 

(0.541) 

(0.499) 

(0.125) 

(0.843) 

(0.910) 

(0.232) 

(0.787) 

Positive  Social 

0.05 

-0.01 

0.463 

0.07 

-0.22A 

0.01 

0.190 

0.12 

0.02 

-0.09 

0.444 

Behavior-TRS  (+) 

(0.357) 

(0.831) 

(0.439) 

(0.083) 

(0.954) 

(0.183) 

(0.853) 

(0.582) 

Problem  Behavior- 

0.04 

-0.02 

0.374 

0.04 

0.09 

-0.09 

0.229 

0.20** 

0.04 

-0.10 

0.028* 

CR  (-) 

(0.477) 

(0.748) 

(0.751) 

(0.323) 

(0.287) 

(0.008) 

(0.658) 

(0.236) 

Problem  Behavior- 

-0.03 

0.02 

0.483 

-0.04 

-0.12A 

-0.04 

0.983 

0.00 

-0.14A 

-0.07 

0.898 

PCR  (-) 

(0.587) 

(0.759) 

(0.571) 

(0.094) 

(0.425) 

(0.974) 

(0.074) 

(0.275) 

Problem  Behavior- 

0.00 

0.02 

0.752 

-0.05 

0.12 

-0.08 

0.184 

-0.03 

0.02 

-0.14 

0.349 

TRS  (-) 

(0.962) 

(0.784) 

(0.589) 

(0.173) 

(0.512) 

(0.732) 

(0.805) 

(0.290) 

ADHD-Related 

-0.02 

-0.01 

0.787 

-0.05 

0.06 

-0.07 

0.474 

-0.09 

-0.01 

-0.08 

0.838 

Behavior-TRS  (-) 

(0.618) 

(0.874) 

(0.599) 

(0.538) 

(0.446) 

(0.295) 

(0.903) 

(0.226) 

See  notes  at  end  of  table. 
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Table  1.35.  Combined-program  results  for  child  outcomes,  by  fidelity  of  implementation— Continued 


to 

CN 


Year  1 

(Spring  3rd  grade) 
Effect  size 
(p-value)  ,:> 


p-value 

for 


Scale-Report 

Academics  Domain 


Year  2 

(Spring  4th  grade) 
Effect  size 
(p-value) 


p-value 

for 

differ- 


Year  3 

(Spring  5th  grade) 
Effect  size 
(p-value) 


Engagement  with 

-0.02 

-0.07 

0.442 

-0.03 

-0.06 

-0.05 

0.772 

-0.18* 

0.06 

-0.02 

0.075' 

Learning-CR  (+) 

(0.670) 

(0.127) 

(0.640) 

(0.410) 

(0.345) 

(0.050) 

(0.406) 

(0.699) 

Academic  Competence 

0.02 

-0.06 

0.225 

-0.04 

0.01 

-0.13* 

0.559 

-0.05 

-0.04 

-0.09 

0.965 

and  Motivation-TRS  (+) 

(0.630) 

(0.226) 

(0.628) 

(0.926) 

(0.033) 

(0.606) 

(0.643) 

(0.261) 

Perceptions  of  School 

Climate  Domain 

Positive  School 

0.02 

0.06 

0.670 

-0.07 

0.04 

0.09 

0.409 

-0.32** 

-0.04 

0.18 

0.026’ 

Orientation-CR  (+) 

(0.795) 

(0.488) 

(0.427) 

(0.804) 

(0.478) 

(0.001) 

(0.732) 

(0.242) 

Negative  School 

-0.03 

-0.06 

0.578 

0.03 

-0.01 

-0.06 

0.457 

0.21* 

-0.02 

-0.19 

0.021’ 

Orientation-CR  (-) 

(0.411) 

(0.231) 

(0.672) 

(0.928) 

(0.485) 

(0.035) 

(0.841) 

(0.102) 

Student  Afraid  at 

-0.04 

-0.05 

0.759 

-0.02 

0.00 

-0.09 

0.675 

0.02 

0.11 

-0.20A 

0.016’ 

School-CR  (-) 

(0.484) 

(0.255) 

(0.780) 

(0.994) 

(0.234) 

(0.823) 

(0.199) 

(0.062) 

Victimization  at 

-0.02 

0.01 

0.700 

-0.04 

-0.12A 

0.00 

0.611 

0.07 

-0.08 

0.01 

0.349 

School-CR  (-) 

(0.632) 

(0.879) 

(0.658) 

(0.083) 

(0.982) 

(0.557) 

(0.209) 

(0.948) 

* Significantly  different  from  zero  at  the  .05  level. 


**  Significantly  different  from  zero  at  the  .01  level. 

A Significantly  different  from  zero  at  the  .10  to  > .05  level,  two-tailed  test. 

1 Test  of  significant  difference  between  Fidelity  subgroups’  effect  sizes. 

NOTE:  Abbreviations  are 
CR:  Child  Report 
PCR:  Primary  Caregiver  Report 
TRS:  Teacher  Report  on  Student 
ADHD:  Attention  deficit  hyperactivity  disorder 

The  +/-  signs  in  parentheses  indicate  the  direction  of  a beneficial  outcome.  All  impact  estimates  were  calculated  using  regression  models  in  which  each  program  and  school  within  a 
program  was  weighted  equally.  The  standard  errors  of  all  estimates  account  for  design  effects  due  to  unequal  weighting  and  the  clustering  of  students  within  schools.  The  effect  size 
was  calculated  by  dividing  the  estimated  impact  by  the  standard  deviation  of  the  outcome  measure  for  the  control  group. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Summary  of  Results  Regarding  Year-by-Year  Impacts 

The  combined-program  analysis  of  the  year-by-year  impacts  did  not  yield  evidence  that  the  seven  SACD 
programs  improved  student  outcomes.  A small  number  of  findings  were  statistically  significant  (but  no  more 
than  expected  by  chance  except  for  several  of  the  subgroup  analyses)  or  substantively  important.  These 
results  were  split  into  similar  numbers  of  beneficial  and  detrimental  impacts.  In  the  majority  of  cases,  the 
results  (both  beneficial  and  detrimental)  occurred  only  in  1 year  and  were  not  replicated  across  the  3 years  of 
the  study. 

Specifically,  2 of  60  estimated  impacts  for  the  seven  SACD  programs  combined  for  the  3 years  were  found 
statistically  significant  (with  beneficial  impacts)  and  none  were  found  to  be  substantively  important.  The  lack 
of  findings  at  the  combined-program  level  was  not  found  to  be  due  to  differences  among  the  individual 
programs  or  such  subgroups  as  males  versus  females,  stayers  versus  new  entrants,  and  students  with  different 
levels  of  initial  risk.  However,  it  should  be  noted  that  the  smaller  sample  sizes  available  for  these  analyses 
reduced  the  ability  to  detect  smaller  statistically  significant  impacts  compared  to  the  analysis  of  all  seven 
programs  together. 

For  the  individual  SACD  programs,  16  statistically  significant  impacts  were  found  over  the  3 years  (9 
beneficial  and  7 detrimental)  versus  21  that  would  be  expected  by  chance.  None  occurred  more  than  once  per 
program.  In  addition,  19  nonsignificant  substantively  important  impacts  were  found  (10  beneficial  and  9 
detrimental).  One  program  had  a substantively  important  beneficial  impact  on  an  outcome  for  more  than  1 
year. 

The  subgroup  analyses  produced  a low  number  of  significant  results,  mixed  impacts  (both  beneficial  and 
detrimental),  and  a lack  of  repeated  significant  results,  providing  little  evidence  of  a relationship  between 
subgroups  and  the  impact  of  the  seven  SACD  programs  considered  together.  The  analysis  by  gender  found 
eight  significant  differences  in  impacts  but  no  pattern  favoring  one  gender.  No  significant  results  were  found 
for  the  analysis  of  stayers  versus  new  entrants.  For  the  analysis  of  five  types  of  different  initial  risk,  41 
significant  marginal  effects  were  found  (versus  13  to  14  expected  by  chance),  of  which  26  showed  more 
beneficial  impacts  for  higher  risk  students  and  15  found  more  detrimental  impacts  for  higher  risk  students  as 
compared  to  lower  risk  students.  Six  of  these  findings  were  repeated  in  at  least  2 different  years. 

Impacts  on  Growth  of  Student  Outcomes 

A growth  curve  analysis  was  done  with  the  3 years  of  data  to  complement  the  three  cross-sectional  impact 
analyses  comparing  the  outcomes  of  students  in  control  and  treatment  schools  at  the  end  of  their  third-grade 
year  (spring  2005),  fourth -grade  year  (spring  2006),  and  fifth -grade  year  (spring  2007).  The  growth  curve 
analysis  examined  the  average  change  over  time  in  the  child  outcomes  between  fall  2004  and  spring  2007. 
Unlike  the  cross-sectional  analysis,  this  was  not  a cumulative  analysis  but  an  analysis  of  the  differences  in 
growth  rates  between  students.  Specifically,  it  examined  the  estimated  impacts  on  the  trajectories  of  student 
outcomes  over  time.  The  growth  curve  analysis  used  the  same  covariates  and  compared  results  across  the 
same  subgroups  as  the  cross-sectional  analyses.  In  comparison  to  the  cross-sectional  analysis,  the  growth 
curve  approach  used  all  available  data  and  multiple  measures  of  the  outcomes  because  students  at  any  survey 
point  contributed  to  the  impact  estimates. 

The  growth  curve  analysis  addressed  a subset  of  the  research  questions  regarding  the  SACD  evaluation 
(combined-program  and  individual  program).  These  included  the  following: 

1 . What  were  the  average  impacts  of  the  seven  SACD  programs  together  on  the  average  growth  over 
time  in  student  outcomes  during  the  study  period,  from  implementation  of  the  programs  in  fall  2004 
through  spring  2007? 
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2.  What  were  the  impacts  on  the  average  growth  over  time  in  student  outcomes  during  the  study  period 
by  individual  SACD  program? 

3.  Did  the  impacts  on  the  average  growth  over  time  in  student  outcomes  during  the  study  period  differ 
by  gender,  stayer  versus  new  entrant  status,  and  initial  risk  factors? 

As  in  the  case  of  the  cross-sectional  year-by-year  analysis,  the  combined  analysis  used  for  the  first  question, 
with  its  larger  sample  size  and  associated  greater  power,  would  be  able  to  detect  smaller  statistically  significant 
impacts  than  the  analyses  for  the  second  and  third  questions. 

Sample 

The  sample  of  students  for  the  growth  curve  analysis  included  all  students  who  were  enrolled  in  one  of  the 
study  schools  during  the  study  period,  who  were  eligible  to  complete  a survey  at  the  initial  data  collection  or 
at  any  of  the  follow-up  survey  points,  and  who  had  data  from  one  or  more  data  collection  points.40  Student 
outcomes  were  measured  at  five  time  points.  Initial  measures  were  collected  in  fall  2004,  and  follow-up 
measures  were  collected  in  spring  2005,  fall  2005,  spring  2006,  and  spring  2007.  The  outcomes  included  the 
18  child-level  outcome  measures  across  the  four  outcome  domains  (Social  and  Emotional  Competence, 
Behavior,  Academics,  and  Perceptions  of  School  Climate).  Two  additional  outcomes,  Feelings  of  Safety  and 
Student  Support  for  Teacher,  were  used  in  the  cross-sectional  analysis  but  were  not  used  in  the  growth  curve 
model  because  they  were  measured  at  the  teacher  level. 

Table  1.36  displays  the  sample  sizes  and  percentages  of  sample  with  a response  across  the  three  reports 
(Child,  Primary  caregiver,  and  Teacher).  The  sample  sizes  and  percentages  varied  across  survey  instruments 
because  of  differences  in  survey  consent  and  completion  rates.  At  each  survey  period,  the  teachers  were  most 
likely  to  provide  data  on  children  (the  percentage  of  the  sample  universe  with  responses  ranged  from  63%  to 
66%).  The  primary  caregivers  were  least  likely  to  provide  data  on  children  (the  percentage  of  the  sample 
universe  with  responses  ranged  from  46%  to  58%).  The  percentage  of  the  sample  with  responses  was  similar 
for  treatment  and  control  schools  in  most  survey  waves.  However,  it  was  statistically  significantly  higher  for 
the  treatment  group  for  the  Child  Report  and  Primary  Caregiver  Report  in  fall  2004  and  for  the  Child  Report 
in  spring  2005.  Table  1.36  also  shows  the  percentages  of  the  follow-up  sample  members  who  were  in  the 
initial  sample.  There  was  considerable  turnover,  as  evidenced  by  the  decreasing  percentage  over  time  of 
students  who  were  in  the  initial  fall  2004  sample.  By  spring  2007,  across  the  three  survey  instruments,  about 
two-thirds  of  the  sample  had  been  part  of  the  original  fall  2004  survey  (66%  for  the  Child  Report  and  the 
Primary  Caregiver  Report,  and  68%  for  the  Teacher  Report  on  Student).  There  were  no  statistically  significant 
differences  in  the  level  of  turnover  by  treatment  status. 


40  Sensitivity  analyses  discussed  later  in  this  section  examined  the  robustness  of  the  findings  when  using  smaller  samples 
of  students  who  had  more  rounds  of  data. 
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Table  1.36.  Growth  curve  analysis  sample  size,  percentage  of  sample  universe,  and  percentage  of  initial  sample 


Analysis  sample  size 

(Percentage  of  sample  universe) 

Child  Report Primary  Caregiver  Report  Teacher  Report  on  Student 


T reat-  T reat-  T reat- 


Program 

ment 

Control 

Total 

ment 

Control 

Total 

ment 

Control 

Total 

Fall  2004  Overall 

2,092 

1,905 

3,997 

1,980 

1,794 

3,774 

2,157 

1,946 

4,103 

(62.1)** 

(59.5) 

(60.9) 

(58.8)** 

(56.1) 

(57.5) 

(64.1) 

(60.8) 

(62.5) 

Spring  2005 

Overall 

2,200 

1,967 

4,167 

1,768 

1,613 

3,381 

2,264 

2,029 

4,293 

(64.9)*** 

(61.3) 

(63.2) 

(52.2) 

(50.3) 

(51.3) 

(66.8) 

(63.2) 

(65.1) 

Percentage  of  sample  in  fall  2004 

86.4 

88.0 

87.1 

88.0 

88.5 

88.2 

89.7 

89.9 

89.8 

Fall  2005 

Overall 

2,145 

1,952 

4,097 

1,761 

1,623 

3,384 

2,189 

1,963 

4,152 

(65.4) 

(64.8) 

(65.1) 

(53.7) 

(53.8) 

(53.8) 

(66.7) 

(65.1) 

(66.0) 

Percentage  of  sample  in  fall  2004 

74.9 

76.3 

75.6 

74.4 

75.9 

75.1 

77.7 

77.7 

77.7 

Spring  2006 

Overall 

2,146 

1,926 

4,072 

1,682 

1,517 

3,199 

2,239 

2,014 

4,253 

(64.5) 

(62.4) 

(63.5) 

(50.6) 

(49.1) 

(49.9) 

(67.3) 

(65.2) 

(66.3) 

Percentage  of  sample  in  fall  2004 

71.6 

73.8 

72.6 

73.3 

74.2 

73.7 

74.4 

75.4 

74.9 

Spring  2007 

Overall 

2,059 

1,935 

3,994 

1,454 

1,414 

2,875 

2,095 

1,994 

4,089 

(64.9) 

(62.9) 

(63.9) 

(46.1) 

(46.0) 

(46.0) 

(66.0) 

(64.8) 

(65.4) 

Percentage  of  sample  in  fall  2004 

66.0 

65.6 

65.8 

66.6 

64.7 

65.6 

68.7 

67.4 

68.0 

**  Treatment  group  significantly  different  from  control  group  at  the  .01  level. 

***  Treatment  group  significantly  different  from  control  group  at  the  .001  level. 

NOTE:  This  table  is  based  on  categorizing  students  according  to  their  current  treatment  status  at  a point  in  time.  During  the  course  of  the  study  about  2 percent  of  students  changed 
status  when  they  moved  from  a treatment  to  a control  school  (or  vice  versa).  These  students  were  assigned  to  their  baseline  treatment  status  to  preserve  the  random  assignment 
design. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Table  1.37  shows  the  average  scores  for  each  of  the  18  outcome  measures  by  treatment  status  and  data 
collection  point.  All  students  with  a response  for  an  outcome  measure  are  included  in  the  data,  regardless  of 
when  they  entered  the  study  schools.  Because  all  students  are  included,  the  treatment-control  differences 
reflect  both  behavioral  changes  within  students  who  have  data  from  more  than  one  survey  and  compositional 
changes  in  the  children  for  whom  data  were  collected.  These  compositional  changes  were  due  to  the  mobility 
of  the  sample,  and  table  1.36  shows  little  evidence  of  differential  mobility  by  treatment  status.  Table  1.37 
shows  that  the  outcome  measures  followed  very  similar  trajectories  for  the  students  in  the  treatment  and 
control  schools  (as  described  in  detail  later  in  this  section).  This  holds  true  across  domains  and  across  the 
three  data  reports. 
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Table  1 .37.  Child-level  outcomes,  by  survey  period  and  treatment  group  status 

Child-level  outcome 

Fall  Spring  Fall  Spring  Spring 

Average  outcome  of  children  with  data  in  any  wave1 2004 2005  2005  2006  2007 

Social  and  Emotional  Competence  Domain 
Self-Efficacy  for  Peer  Interaction-(CR)  (+) 


Control  group 

2.95 

3.07 

3.16 

3.22 

3.24 

Treatment  group 

2.93 

3.04 

3.14 

3.18* 

3.22 

Normative  Beliefs  About  Aggression-(CR)  (-) 

Control  group 

1.23 

1.30 

1.30 

1.36 

1.45 

Treatment  group 

1.22 

1.29 

1.32 

1.36 

1.46 

Empathy-(CR)  (+) 

Control  group 

2.41 

2.30 

2.27 

2.19 

2.10 

Treatment  group 

2.42 

2.32 

2.25 

2.17 

2.08 

Behavior  Domain 

Altruistic  Behavior-(CR)  (+) 

Control  group 

1.45 

1.28 

1.17 

1.09 

1.05 

Treatment  group 

1.45 

1.24 

1.14 

1.05 

1.05 

Altruistic  Behavior-(PCR)  (+) 

Control  group 

2.29 

2.23 

2.23 

2.23 

2.26 

Treatment  group 

2.28 

2.27 

2.23 

2.25 

2.26 

Altruistic  Behavior-(TRS)  (+) 

Control  group 

1.36 

1.38 

1.29 

1.34 

1.36 

Treatment  group 

1.40 

1.44 

1.34* 

1.37 

1.34 

Positive  Social  Behavior-(PCR)  (+) 

Control  group 

3.00 

3.03 

3.07 

3.06 

3.08 

Treatment  group 

3.00 

3.02 

3.06 

3.07 

3.09 

Positive  Social  Behavior-(TRS)  (+) 

Control  group 

3.05 

3.02 

3.01 

3.06 

3.10 

Treatment  group 

3.04 

3.03 

3.02 

3.01 

3.07 

Problem  Behavior-(CR)  (-) 

Control  group 

0.23 

0.29 

0.28 

0.34 

0.46 

Treatment  group 

0.24 

0.30 

0.31 

0.36 

0.50 

Problem  Behavior-(PCR)  (-) 

Control  group 

1.57 

1.57 

1.55 

1.55 

1.55 

Treatment  group 

1.56 

1.55 

1.53 

1.53 

1.53 

Problem  Behavior-(TRS)  (-) 

Control  group 

1.35 

1.45 

1.38 

1.44 

1.46 

Treatment  group 

1.35 

1.46 

1.38 

1.45 

1.46 

ADHD-Related  Behavior-(TRS)  (-) 

Control  group 

1.72 

1.74 

1.72 

1.69 

1.68 

Treatment  group 

1.73 

1.75 

1.69 

1.69 

1.67 

See  notes  at  end  of  table. 
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Table  1.37.  Child-level  outcomes,  by  survey  period  and  treatment  group  status — Continued 


Child-level  outcome 


Average  outcome  of  children  with  data  in  any  wave1 

Fall 

2004 

Spring 

2005 

Fall 

2005 

Spring 

2006 

Spring 

2007 

Academics  Domain 

Engagement  with  Learning-(CR)  (+) 

Control  group 

3.70 

3.68 

3.72 

3.69 

3.61 

Treatment  group 

3.66 

3.66 

3.70 

3.66 

3.58 

Academic  Competence  and  Motivation-(TRS)  (+) 

Control  group 

2.95 

3.02 

2.93 

3.01 

3.02 

Treatment  group 

2.89 

2.96 

2.89 

2.93 

2.94 

Perceptions  of  School  Climate  Domain 
Positive  School  Orientation-(CR)  (+) 

Control  group 

3.23 

2.86 

2.81 

2.64 

2.56 

Treatment  group 

3.21 

2.86 

2.82 

2.65 

2.51 

Negative  School  Orientation-(CR)  (-) 

Control  group 

1.82 

1.90 

1.87 

1.97 

2.09 

Treatment  group 

1.87 

1.89 

1.87 

1.98 

2.10 

Student  Afraid  at  School-(CR)  (-) 

Control  group 

2.38 

2.30 

2.22 

2.25 

2.21 

Treatment  group 

2.43 

2.27 

2.21 

2.22 

2.20 

Victimization  at  School-(CR)  (-) 

Control  group 

0.76 

0.77 

0.65 

0.69 

0.73 

Treatment  group 

0.77 

0.75 

0.64 

0.66 

0.73 

* Treatment  group  significantly  different  from  control  group  at  the  .05  level. 

1 Average  outcome  is  the  average  unadjusted  score  on  the  outcome  measure  across  all  students  in  that  wave  of  data. 

NOTE:  Abbreviations  are 
CR:  Child  Report 
PCR:  Primary  Caregiver  Report 
TRS:  Teacher  Report  on  Student 
ADHD:  Attention  deficit  hyperactivity  disorder 

The  +/-  signs  in  parentheses  indicate  the  direction  of  a beneficial  outcome.  All  impact  estimates  were  calculated  using  regression 
models  in  which  each  program  and  school  within  a program  was  weighted  equally.  The  standard  errors  of  all  estimates  account  for 
design  effects  due  to  unequal  weighting  and  the  clustering  of  students  within  schools.  The  effect  size  was  calculated  by  dividing  the 
estimated  impact  by  the  standard  deviation  of  the  outcome  measure  for  the  control  group.  This  table  is  based  on  categorizing 
students  according  to  their  current  treatment  status  at  a point  in  time.  During  the  course  of  the  study  about  2 percent  of  students 
changed  status  when  they  moved  from  a treatment  to  a control  school  (or  vice  versa).  These  students  were  assigned  to  their 
baseline  treatment  status  to  preserve  the  random  assignment  design.  The  majority  of  the  outcomes  (13)  were  on  a scale  of  1 -4.  The 
scales  for  the  others  were  as  follows:  Empathy  1 -3,  Altruistic  Behavior  (CR)  0-3,  Problem  Behavior  (CR)  0-3,  Academic 
Competence  and  Motivation  1-5,  and  Victimization  at  School  0-3. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Analysis 

The  Growth  Curve  Model 

The  SACD  programs’  impacts  over  time  were  estimated  using  growth  curve  models  by  examining  treatment 
and  control  group  differences  in  the  trajectories  of  student  outcomes  during  the  follow-up  period  while 
accounting  for  clustering  at  the  school  level.  The  growth  curve  models  were  estimated  using  a three-level 
hierarchical  linear  model,  where  Level  1 corresponds  to  time,  Level  2 to  students,  and  Level  3 to  schools: 

Level  1 Model — Time 


(1)  yds  — Go  is  ~h  Cl  lis  (TSItis)  ~h  Utis 

where  ytis  is  the  outcome  variable  of  student  i in  school  s at  time  /,  TSItis  is  the  time  measured  in  years 
between  implementation  of  the  SACD  program  and  each  survey  date,41  Uus  is  a mean  zero  disturbance  term, 
aos  and  ai,s  are  random  student-level  intercepts  and  slopes,  respectively,  and  the  program-level  subscript  p is 
omitted  for  simplicity.  The  growth  curve  models  estimate  the  effect  of  the  SACD  programs  (together  and 
separately)  on  outcomes  measured  near  the  start  of  the  implementation,  and  then  at  the  time  of  the  four 
follow-up  data  points  (spring  2005,  fall  2005,  spring  2006,  and  spring  2007). 

Level  2 Model — Students 


(2)  Go  is  — Ps  ~h  XisO  T Sois 
(3)  G lis  — 8s  ~h  Glis 

where  Xis  is  a vector  of  student-level  characteristics  measured  initially  (the  covariates  used  in  the  cross- 
sectional  analyses),  Eos  and  Eus  are  mean  zero  disturbance  terms,  /?,  and  8S  are  random  school-level  coefficients, 
and  0 is  a vector  of  parameters  to  be  estimated.  The  age  of  the  child  at  the  initial  data  collection  was  included 
as  a control  in  the  model. 

Level  3 Model — Schools 


(4)  ps  — ho  ~h  hiTs  ~h  (f)sZ  + Tos 
(5)  8s  = yo  + yiTs  + tis 

where  Ts  is  an  indicator  of  whether  the  school  is  in  the  treatment  group,  <j)s  are  program-specific  fixed  effects 
(Z  indicates  each  of  the  seven  SACD  programs),  x os  and  in  are  mean  zero  disturbance  terms  (that  are  assumed 
to  be  correlated  with  each  other  and  with  the  error  term  in  equations  (4)  and  (5)  for  the  same  school,  but  not 
across  schools),  and  yo,  yi,  "ko,  and  ki  are  parameters  to  be  estimated. 

When  the  student-  and  school-level  equations  are  substituted  into  equation  (5),  this  results  in  the  following 
unified  model: 

(6)  ytis  — 2-o  8-  ?-i  Ts  + XisO  + (j)sZ  + yo(TSItis)  + yi  Ts(TSItiS)  + error 

where  error  = Uus  + Sois  + Tos  + Siis(TSItis)  + Tis(TSIhs).  In  this  formulation,  the  estimate  of  the  slope,  yi, 
represents  the  treatment  and  control  group  difference  in  the  mean  growth  of  the  outcome  measure  between 


41  Time  of  implementation  was  set  at  the  program  level.  These  dates  were  used  both  in  treatment  and  control  schools 
within  that  program.  If  the  survey  collection  date  for  a student  was  missing  in  a round  of  data,  it  was  imputed  using  the 
average  date  within  the  school  for  that  round  of  data  collection. 
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the  first  and  last  measurement  points.  The  estimate  of  "Ki  represents  the  impact  at  the  start  of  the  evaluation 
and  was  expected  not  to  differ  significandy  from  zero.  A linear  specification  of  the  time  metric  is  used 
because  the  data  indicate  that  the  change  in  the  outcomes  followed  a linear  trajectory  over  time.42 

The  growth  curve  analysis  included  the  same  set  of  covariates — X in  equation  (6) — that  were  used  for  the 
cross-sectional  analysis  (table  1.24)  to  ensure  the  comparability  of  the  results.  The  only  exception  was  the 
exclusion  of  the  initial  outcome  measure  because  it  was  used  as  the  outcome  measure  for  the  growth  curve 
analysis  at  time  1 (fall  2004).  As  in  the  cross-sectional  analysis,  missing  covariates  were  imputed  using  mean 
values  for  nonmissing  cases,  by  school,  gender,  race,  and  ethnicity.  Similarly,  sample  weights  were  used  in  all 
analyses  to  (1)  give  each  program  equal  weight  in  the  calculation  of  overall  impact  estimates,  and  (2)  give  each 
school  equal  weight  in  each  program.  However,  the  weights  were  not  adjusted  for  consent  and  response 
differences  across  classrooms  or  schools  because  the  population  of  students  within  the  schools  changed  over 
time  as  students  entered  and  left  the  schools.  The  weights  were  also  constructed  to  give  equal  weight  to  each 
time  period. 

Equation  (6)  was  estimated  for  all  programs  together  and  for  each  program  separately.  In  the  program-level 
models,  the  program-specific  fixed  effects  were  removed  from  the  model  by  dropping  Z.  The  key  impact 
parameter  in  the  growth  curve  model  in  equation  (6)  is  yi,  which  represents  the  extent  to  which  the  growth  in 
the  outcome  during  the  follow-up  period  differed  between  the  treatment  and  control  groups.  The  estimated 
intercept  parameter  (yo)  is  of  interest  because  it  signifies  a model-generated  mean  initial  value  of  the  outcome, 
and  thus  provides  a base  for  interpreting  the  slope  parameter.  The  model  generated  the  following  impact 
information  for  each  child  outcome:  (1)  the  average  intercept  value  for  the  treatment  and  control  groups  (Xo 
+ pki  +J0,  where  p is  the  percentage  of  the  sample  assigned  to  the  treatment  group);  (2)  the  estimated 
growth  for  the  treatment  group  (yi  + yo);  (3)  the  estimated  growth  for  the  control  group  (yo);  and  (4)  the 
estimated  impact  of  the  SACD  programs  (and  each  program  individually)  on  growth  (yi). 

Growth  curve  effect  sizes  were  calculated  by  dividing  the  estimated  impact  of  the  treatment  on  the  outcome 
growth  trajectory  by  the  standard  deviation  of  that  outcome  (which  was  calculated  using  the  initial  data  for 
the  weighted  control  group).  One  year  was  the  unit  of  time  used  in  the  growth  curve  analysis,  so  the 
estimated  impact  of  the  treatment  on  the  growth  trajectory  equals  the  difference  between  the  treatment 
group’s  outcome  and  the  control  group’s  outcome,  on  average,  after  one  year.43  After  dividing  by  the  standard 
deviation  of  the  control  group  outcome,  the  effect  size  measures  the  number  of  standard  deviations  the 
treatment  group  differs  from  the  control  group  after  one  average  year  of  the  SACD  program.  This  approach 
was  analogous  to  the  effect  size  calculations  for  the  cross-sectional  analysis.  The  primary  difference  between 
the  effect  size  for  the  cross-sectional  analysis  and  the  effect  size  for  the  growth  curve  trajectory  is  the  time 
unit  used  in  calculating  the  numerator.  In  the  cross-sectional  analysis,  the  numerator  of  the  effect  size  is  the 
estimated  impact  of  the  treatment  on  the  outcome  over  the  time  period  between  the  initial  and  the  follow-up 
survey  period.  In  the  growth  curve  effect  size,  the  numerator  is  the  difference  between  the  treatment  and  the 
control  group  after  one  year  of  the  study,  on  average. 

The  growth  curve  analysis  was  extended  to  examine  differences  by  subgroups.  For  the  subgroup  analyses,  the 
interaction  terms  between  treatment  status,  time  since  implementation,  and  indicators  of  membership  in 
subgroups  were  included  in  equation  (6).  This  produced  an  estimate  of  the  slope  for  subgroup  members  in 
the  treatment  group,  relative  to  subgroup  members  in  the  control  group.  The  same  subgroups  were  used  in 


42  Three  sensitivity  tests  were  done.  First  the  assumption  of  linearity  was  tested  through  the  inclusion  of  a TSI-squared 
term.  Second,  the  potential  for  learning  loss  as  a result  of  summer  school  breaks  was  modeled  by  including  a time 
indicator  that  allowed  the  growth  coefficient  to  decline  (or  increase)  during  the  summer.  Third,  child’s  age  was  used  in 
place  of  time  since  implementation  to  model  growth.  Child’s  age  captures  both  the  age  of  the  child  at  the  initial  data 
collection  and  the  time  since  implementation  of  the  program  in  one  variable. 

43  This  impact  estimate  takes  into  account  differences  between  the  initial  levels  of  the  outcome  for  the  treatment  and 
control  groups,  differences  in  their  covariates,  and  the  effects  of  clustering  at  the  school  level. 
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the  growth  curve  analysis  as  were  used  in  the  cross-sectional  analyses,  and  these  included  (1)  student  gender; 
(2)  new  entrants  versus  members  of  the  original  sample  (stayers);  and  (3)  levels  of  initial  risk  for 
socioeconomic  risk,  family  risk,  perceptions  of  community  risk,  and  child  behavior  risk. 

Results 

None  of  the  18  estimated  impacts  on  the  trajectories  of  child  outcomes  from  the  seven  SACD  programs 
when  combined  were  statistically  significant  (table  1.38).  The  estimated  effect  sizes  were  at  or  below  .07 
(absolute  value).44 

Estimated  impacts  could  differ  across  programs  if  some  SACD  programs  were  more  effective  than  others,  or 
if  the  types  of  children  or  settings  in  some  programs  were  more  conducive  to  positive  program  effects  than 
those  found  in  others.  Therefore,  it  is  possible  that  there  were  no  statistically  significant  impact  estimates  at 
the  overall  level  because  beneficial  impacts  in  some  programs  were  offset  by  negative  or  null  impacts  in 
others.  To  investigate  this  possibility,  the  impacts  at  the  program  level  were  tested. 

Table  1.39  lists  the  outcomes  on  which  each  individual  program  had  statistically  significant  impacts  (a 
program’s  impacts  on  all  the  outcomes  are  given  in  the  program -specific  chapters).  The  results  indicate  that 
the  lack  of  significant  impacts  in  the  overall  evaluation  reflected  the  lack  of  significant  impacts  at  the  program 
level.  Six  impacts  would  be  expected  to  appear  significant  by  chance  at  the  5 percent  significance  level,  given 
that  126  impact  estimates  were  made.  Table  1.39  shows  that  there  were  6 significant  outcomes  found:  2 with 
beneficial  impacts  on  growth  and  4 with  detrimental  impacts  on  growth.  These  outcomes,  and  the  programs 
with  which  they  were  associated,  follow.  CSP  showed  a significant  beneficial  impact  on  growth  for 
Victimization  at  School  (Child  Report)  with  an  effect  size  of  -.09.  PATHS  showed  a significant  beneficial 
impact  on  growth  for  Academic  Competence  (Teacher  Report)  with  an  effect  size  of  0.09.  LBW  showed  a 
significant  detrimental  impact  on  growth  for  Positive  School  Orientation  (Child  Report)  with  an  effect  size  of 
-0.13.  SS  showed  three  significant  detrimental  impacts  on  growth:  for  Engagement  with  Learning  (Child 
Report)  with  an  effect  size  of  -0.09,  Positive  Social  Behavior  (Teacher  Report)  with  an  effect  size  of  -0.19, 
and  Empathy  (Child  Report)  with  an  effect  size  of  -0.13. 


44  When  a squared  value  of  the  time  since  implementation  was  added  to  the  sensitivity  analysis,  three  of  the  outcomes 
(child-reported  Altruistic  Behavior,  child-reported  Victimization,  and  child-reported  Student  Afraid  at  School),  showed  a 
significant  treatment  effect  on  the  growth  trajectory.  These  results  indicated  that  for  child-reported  Altruistic  Behavior, 
the  treatment  had  a short-term  detrimental  impact  that  converged  with  the  control  group  over  time.  For  child-reported 
Victimization  at  School  and  Student  Afraid  at  School,  the  treatment  had  a short-term  beneficial  impact  that  converged 
with  the  control  group  over  time.  Because  there  were  3 significant  results  out  of  18  and  the  direction  of  the  effects  was 
mixed,  the  linear  specification  of  the  growth  curve  model  was  retained. 
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Average  growth  in  the  score  per  year1 

Scale-Report 

Mean  score  at 
implementation2 

Treatment 

group 

Control 

group 

Impact  on 
growth3 

Effect 

4 

size 

Standard 
error  of 
impact 

p-value  of 
impact 

Social  and  Emotional  Competence  Domain 

Self-Efficacy  for  Peer  Interactions-CR  (+) 

2.95 

0.13 

0.13 

0.00 

0.00 

0.01 

0.942 

Normative  Beliefs  About  Aggression-CR  (-) 

1.23 

0.10 

0.07 

0.02 

0.04 

0.01 

0.115 

Empathy-CR  (+) 

2.41 

-0.14A 

-0.12 

-0.02 

-0.05 

0.01 

0.070 

Behavior  Domain 

Altruistic  Behavior-CR  (+) 

1.41 

-0.17 

-0.16 

-0.01 

-0.01 

0.02 

0.681 

Altruistic  Behavior-PCR  (+) 

2.32 

-0.03 

-0.03 

0.01 

0.01 

0.01 

0.616 

Altruistic  Behavior-TRS  (+) 

1.40 

-0.04 

0.00 

-0.04 

-0.07 

0.03 

0.224 

Positive  Social  Behavior-PCR  (+) 

2.99 

0.04 

0.03 

0.01 

0.02 

0.01 

0.217 

Positive  Social  Behavior-TRS  (+) 

3.00 

0.00 

0.01 

-0.01 

-0.01 

0.03 

0.671 

Problem  Behavior-CR  (-) 

0.24 

0.10 

0.08 

0.01 

0.02 

0.01 

0.351 

Problem  Behavior-PCR  (-) 

1.58 

-0.01 

-0.01 

0.00 

0.00 

0.01 

0.843 

Problem  Behavior-TRS  (-) 

1.38 

0.05 

0.05 

0.00 

0.00 

0.01 

0.909 

ADHD-Related  Behavior-TRS  (-) 

1.75 

-0.02 

-0.01 

-0.01 

-0.01 

0.02 

0.707 

Academics  Domain 

Engagement  with  Learning-CR  (+) 

3.69 

-0.03 

-0.03 

0.00 

-0.01 

0.01 

0.707 

Academic  Competence  and  Motivation-TRS  (+) 

2.87 

0.02 

0.03 

-0.01 

-0.01 

0.02 

0.590 

See  note  at  end  of  table. 
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Table  1.38.  Impacts  on  growth  of  child  outcomes  from  combined-program  analysis — Continued 


Average  growth  in  the  score  per  year1 

Standard 


Scale-Report 

Mean  score  at 
implementation2 

Treatment 

group 

Control 

group 

Impact  on 
growth3 

Effect 

4 

size 

error  of 
impact 

p-value  of 
impact 

Perceptions  of  School  Climate  Domain 
Positive  School  Orientation-CR  (+) 

3.09 

-0.24 

-0.21 

-0.03 

-0.03 

0.02 

0.163 

Negative  School  Orientation-CR  (-) 

1.84 

0.10 

0.09 

0.01 

0.01 

0.02 

0.696 

Student  Afraid  at  School-CR  (-) 

2.38 

-0.08 

-0.08 

0.00 

0.00 

0.02 

0.956 

Victimization  at  School-CR  (-) 

0.76 

-0.03 

-0.02 

-0.01 

-0.01 

0.02 

0.368 

A Treatment  group  significantly  different  from  control  group  at  the  .10  to  > .05  level. 

1 Significantly  different  from  zero  at  the  .05  level. 

2 The  average  score  at  implementation  is  calculated  across  treatment  and  control  groups,  using  regression  models  for  adjustment  on  covariates. 

3 Estimated  difference  between  the  slope  of  the  treatment  and  control  groups. 

4 The  slope  of  the  treatment  group  minus  the  slope  of  the  control  group  divided  by  the  standard  deviation  of  the  outcome  for  the  multiprogram  control  group  (the  standard  deviation  is 
calculated  without  accounting  for  school-level  clustering  or  regression  adjustments). 

NOTE:  Abbreviations  are 
CR:  Child  Report 
PCR:  Primary  Caregiver  Report 
TRS:  Teacher  Report  on  Student 
ADHD:  Attention  deficit  hyperactivity  disorder 

The  +/-  signs  in  parentheses  indicate  the  direction  of  a beneficial  outcome.  No  findings  were  found  statistically  significant  at  or  below  the  .05  level.  All  impact  estimates  were  calculated 
using  HLM  6.06.  Sample  weights  were  used  in  all  analyses  to  (1)  give  each  program  equal  weight  within  each  time  period,  (2)  give  each  school  equal  weight  in  each  program  (within 
each  time  period),  and  (3)  give  each  time  period  equal  weight  in  the  analysis. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Table  1.39.  Significant  impacts  from  the  growth  curve  analyses  of  the  individual  programs 


Program 

Significant1  beneficial  impacts 
(Report)  (Effect  size)  (p-value) 

Significant1  detrimental  impacts 
(Report)  (Effect  size)  (p-value) 

Total 

2 

4 

ABC 

CSP 

Victimization  at  School 
(CR)  (-.09)  (.050) 

LBW 

Positive  School  Orientation 
(CR)  (-.13)  (.016) 

PA 

PATHS 

Academic  Competence 
(TRS)  (.08)  (.048) 

4Rs 

SS 

Engagement  with  Learning 
(CR)  (-.09)  (.021) 
Positive  Social  Behavior 
(TRS)  (-.19)  (.019) 
Empathy 

(CR)  (-0.13)  (.028) 

1 Significantly  different  from  zero  at  the  .05  level. 
NOTE:  Abbreviations  are 


ABC:  Academic  and  Behavioral  Competencies  Program 
CSP:  Competence  Support  Program 
LBW:  Love  In  a Big  World 
PA:  Positive  Action 

PATHS:  Promoting  Alternative  Thinking  Strategies 

4Rs:  The  4Rs  Program  (Reading,  Writing,  Respect,  and  Resolution) 

SS:  Second  Step 
CR:  Child  Report 
TRS:  Teacher  Report  on  Student 
Blank  cell:  Finding  of  no  impact 

Out  of  the  126  comparisons  made  (7  programs  times  18  outcomes),  6 would  be  expected  to  be  statistically  significant  at  the  .05 
level  by  chance.  The  number  of  results  found  significant  was  no  more  than  expected  by  chance.  All  impact  estimates  were 
calculated  using  HLM  6.06.  Sample  weights  were  used  in  all  analyses  to  (1)  give  each  program  equal  weight  within  each  time 
period,  (2)  give  each  school  equal  weight  in  each  program  (within  each  time  period),  and  (3)  give  each  time  period  equal  weight  in 
the  analysis. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 


The  subgroup  analyses  found  little  evidence  regarding  differential  impacts  of  the  seven  SACD  programs, 
combined,  on  the  subgroups’  growth  in  outcomes.  For  the  gender  subgroup  analysis,  none  of  the  36 
estimated  impacts  was  statistically  significant  (table  1.40).  For  the  new  entrants  analysis,  1 impact  (on 
Negative  School  Orientation)  was  found  to  differ  significantly  and  detrimentally  for  new  entrants  versus  the 
impact  on  members  of  the  original  sample  (table  1.41).  From  the  18  tests  done,  1 significant  impact  would  be 
expected  by  chance  for  the  new  entrants  analysis. 
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Table  1.40.  Impacts  on  growth  of  child  outcomes,  by  gender,  from  combined-program  analysis 


Boys 

Girls 

Scale-Report 

Impact  on 
annual 
growth 

Effect 

size1 

p-value 

Impact  on 
annual 
growth 

Effect 

size1 

p-value 

p-value  for 
test  of  impact 
differences 

Social  and  Emotional  Competence  Domain 

Self-Efficacy  for  Peer  Interactions-CR  (+) 

0.00 

0.01 

0.785 

0.00 

0.00 

0.885 

0.347 

Normative  Beliefs  About  Aggression-CR  (-) 

0.01 

0.02 

0.557 

0.02 

0.03 

0.443 

0.630 

Empathy-CR  (+) 

-0.01 

-0.02 

0.583 

-0.02 

-0.04 

0.282 

0.349 

Behavior  Domain 

Altruistic  Behavior-CR  (+) 

0.01 

0.01 

0.673 

0.00 

0.00 

0.888 

0.153 

Altruistic  Behavior-PCR  (+) 

0.01 

0.01 

0.612 

0.00 

0.00 

0.864 

0.925 

Altruistic  Behavior-TRS  (+) 

-0.04 

-0.06 

0.211 

-0.05 

-0.10 

0.104 

0.379 

Positive  Social  Behavior-PCR  (+) 

0.01 

0.02 

0.446 

0.00 

0.01 

0.687 

0.933 

Positive  Social  Behavior-TRS  (+) 

-0.01 

-0.01 

0.674 

0.00 

0.00 

0.995 

0.519 

Problem  Behavior-CR  (-) 

0.02 

0.03 

0.332 

0.00 

0.01 

0.794 

0.851 

Problem  Behavior-PCR  (-) 

0.00 

-0.01 

0.590 

0.00 

-0.01 

0.498 

0.708 

Problem  Behavior-TRS  (-) 

-0.01 

-0.02 

0.685 

-0.01 

-0.02 

0.426 

0.932 

ADHD-Related  Behavior-TRS  (-) 

-0.03 

-0.04 

0.219 

-0.01 

-0.01 

0.498 

0.293 

Academics  Domain 

Engagement  with  Learning-CR  (+) 

0.01 

0.02 

0.489 

-0.01 

-0.01 

0.581 

0.933 

Academic  Competence  and  Motivation-TRS  (+) 

-0.01 

-0.01 

0.756 

-0.02 

-0.02 

0.291 

0.495 

See  notes  at  end  of  table. 
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Table  1.40.  Impacts  on  growth  of  child  outcomes,  by  gender,  from  combined-program  analysis — Continued 


Scale-Report 

Impact  on 
annual 
growth 

Boys 

Effect 

size1 

p-value 

Impact  on 
annual 
growth 

Girls 

Effect 

size1 

p-value 

p-value  for 
test  of  impact 
differences 

Perceptions  of  School  Climate  Domain 
Positive  School  Orientation-CR  (+) 

-0.01 

-0.01 

0.680 

-0.01 

-0.01 

0.742 

0.571 

Negative  School  Orientation-CR  (-) 

0.00 

0.00 

0.970 

-0.01 

-0.02 

0.386 

0.408 

Student  Afraid  at  School-CR  (-) 

0.00 

0.00 

0.880 

-0.04 

-0.03 

0.149 

0.213 

Victimization  at  School-CR  (-) 

-0.02 

-0.02 

0.205 

0.00 

0.00 

0.930 

0.349 

1 The  slope  of  the  treatment  group  minus  the  slope  of  the  control  group  divided  by  the  standard  deviation  of  the  outcome  for  the  combined-program  control  group  (the  standard 
deviation  is  calculated  without  accounting  for  school-level  clustering  or  regression  adjustments). 

NOTE:  Abbreviations  are 
CR:  Child  Report 
PCR:  Primary  Caregiver  Report 
TRS:  Teacher  Report  on  Student 
ADHD:  Attention  deficit  hyperactivity  disorder 

The  +/-  signs  in  parentheses  indicate  the  direction  of  a beneficial  outcome.  No  findings  were  found  statistically  significant  at  or  below  the  .05  level.  All  impact  estimates  were  calculated 
using  HLM  6.06.  Sample  weights  were  used  in  all  analyses  to  (1)  give  each  program  equal  weight  within  each  time  period,  (2)  give  each  school  equal  weight  in  each  program  (within 
£ each  time  period),  and  (3)  give  each  time  period  equal  weight  in  the  analysis, 
o SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Table  1.41.  Impacts  on  growth  of  child  outcomes  for  new  entrants  versus  original  members,  from  combined-program  analysis 


Original  members 

New  entrants 

Scale-Report 

Impact  on 
annual 
growth 

Effect 

size1 

p-value 

Impact  on 
annual 
growth 

Effect 

size1 

p-value 

p-value  for 
test  of  impact 
differences 

Social  and  Emotional  Competence  Domain 

Self-Efficacy  for  Peer  Interactions-CR  (+) 

0.00 

0.00 

0.947 

0.00 

0.00 

0.918 

0.931 

Normative  Beliefs  About  Aggression-CR  (-) 

0.02 

0.05 

0.256 

0.01 

0.02 

0.725 

0.867 

Empathy-CR  (+) 

-0.02 

-0.05 

0.107 

-0.02 

-0.05 

0.441 

0.475 

Behavior  Domain 

Altruistic  Behavior-CR  (+) 

-0.01 

-0.01 

0.783 

0.01 

0.01 

0.869 

0.251 

Altruistic  Behavior-PCR  (+) 

0.01 

0.01 

0.693 

0.01 

0.01 

0.742 

0.752 

Altruistic  Behavior-TRS  (+) 

-0.05 

-0.11 

0.107 

0.02 

0.05 

0.629 

0.764 

Positive  Social  Behavior-PCR  (+) 

0.01 

0.01 

0.442 

-0.02 

-0.04 

0.350 

0.563 

Positive  Social  Behavior-TRS  (+) 

0.00 

0.00 

0.881 

-0.02 

-0.02 

0.729 

0.254 

Problem  Behavior-CR  (-) 

0.01 

0.01 

0.732 

0.00 

0.01 

0.922 

0.320 

Problem  Behavior-PCR  (-) 

0.00 

0.01 

0.707 

-0.01 

-0.02 

0.629 

0.851 

Problem  Behavior-TRS  (-) 

-0.01 

-0.01 

0.710 

-0.01 

-0.01 

0.838 

0.427 

ADHD-Related  Behavior-TRS  (-) 

-0.02 

-0.02 

0.318 

0.02 

0.02 

0.694 

0.107 

Academics  Domain 

Engagement  with  Learning-CR  (+) 

-0.01 

-0.01 

0.577 

-0.03 

-0.04 

0.228 

0.709 

Academic  Competence  and  Motivation-TRS  (+) 

-0.02 

-0.02 

0.281 

-0.02 

-0.02 

0.739 

0.910 

See  notes  at  end  of  table. 
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Table  1.41.  Impacts  on  growth  of  child  outcomes  for  new  entrants  versus  original  members,  from  combined-program  analysis — 
Continued 


Original  members New  entrants 


Impact  on  Impact  on  p-value  for 

annual  Effect  annual  Effect  test  of  impact 


Scale-Report 

growth 

size 

p-value 

growth 

size 

p-value 

differences 

Perceptions  of  School  Climate  Domain 
Positive  School  Orientation-CR  (+) 

-0.01 

-0.01 

0.592 

-0.05 

-0.06 

0.161 

0.113 

Negative  School  Orientation-CR  (-) 

-0.01 

-0.01 

0.627 

0.05 

0.06A 

0.067 

0.012’ 

Student  Afraid  at  School-CR  (-) 

-0.02 

-0.02 

0.427 

-0.05 

-0.06 

0.304 

0.701 

Victimization  at  School-CR  (-) 

-0.01 

-0.01 

0.435 

0.02 

0.02 

0.568 

0.643 

* Significantly  different  from  zero  at  the  .05  level. 

A Significantly  different  from  zero  at  the  .10  to  > .05  level. 

1 The  slope  of  the  treatment  group  minus  the  slope  of  the  control  group  divided  by  the  standard  deviation  of  the  outcome  for  the  multisite  control  group  (the  standard  deviation  is 
calculated  without  accounting  for  school-level  clustering  or  regression  adjustments). 

NOTE:  Abbreviations  are 
CR:  Child  Report 
PCR:  Primary  Caregiver  Report 
TRS:  Teacher  Report  on  Student 
ADHD:  Attention  deficit  hyperactivity  disorder 

The  +/-  signs  in  parentheses  indicate  the  direction  of  a beneficial  outcome.  All  impact  estimates  were  calculated  using  HLM  6.06.  Sample  weights  were  used  in  all  analyses  to  (1)  give 
each  program  equal  weight  within  each  time  period,  (2)  give  each  school  equal  weight  in  each  program  (within  each  time  period),  and  (3)  give  each  time  period  equal  weight  in  the 
analysis. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Few  of  the  growth  trajectories  differed  across  levels  of  initial  risk  (tables  1.42  through  1.46).  An  example 
illustrates  how  to  interpret  the  tables.  Table  1.42  shows  the  impacts  on  the  growth  trajectory  of  the  outcome 
scales  by  the  child’s  initial  level  of  socioeconomic  risk.  The  first  row  in  the  table  displays  the  estimated 
impacts  on  the  growth  trajectory  of  child-reported  Self-Efficacy  for  Peer  Interactions  at  three  levels  of  initial 
risk:  (1)  at  one  standard  deviation  below  the  mean  risk  level,  (2)  at  the  mean  risk  level,  and  (3)  at  one  standard 
deviation  above  the  mean  risk  level.  When  a child  had  a low  level  of  risk  (one  standard  deviation  below  the 
mean),  the  difference  between  the  growth  trajectories  of  the  treatment  group  and  the  control  group  for  Self- 
Efficacy  for  Peer  Interactions  was  -0.01.  When  a child  had  an  average  level  of  risk  (at  the  mean),  the 
difference  between  the  treatment  group  and  the  control  group  was  -0.01.  When  a child  had  a high  level  of  risk 
(one  standard  deviation  above  the  mean),  the  difference  between  the  treatment  group  and  the  control  group 
was  0.00.  The  interaction  term  showed  that  with  each  one-unit  increase  in  socioeconomic  risk,  the  impact  on 
the  Self-Efficacy  for  Peer  Interactions  trajectory  increased  by  0.01.  In  other  words,  the  higher  the  child’s 
initial  level  of  socioeconomic  risk,  the  more  beneficial  was  the  SACD  program  for  the  child’s  growth  in  Self- 
Efficacy  for  Peer  Interactions.  However,  this  difference  in  the  SACD  programs’  impact  across  risk  levels  was 
not  statistically  significantly  different  from  zero  (interaction  term  = 0.01;  not  significant  at  the  .05  level). 

There  were  6 outcomes  out  of  90  estimated  impacts  for  which  growth  differed  significantly  across  initial  risk 
levels  (versus  4 to  5 expected  by  chance).  For  4 outcomes,  the  seven  SACD  programs  combined  had  more 
beneficial  impacts  on  growth  for  children  with  higher  initial  risk  levels.  The  SACD  programs  resulted  in 
greater  growth  of  parent-reported  Altruistic  Behavior  for  children  with  higher  levels  of  socioeconomic  risk 
(interaction  term  = 0.03  on  table  1.42);  greater  growth  on  Academic  Competence  and  less  growth  of  child- 
reported  Victimization  at  School  among  children  with  higher  levels  of  family  risk  (interaction  term  = 0.13  and 
-0.13,  respectively,  on  table  1.43);  and  less  growth  of  child-reported  Victimization  at  School  among  children 
with  higher  levels  of  perceived  community  risk  (interaction  term  = -0.06  on  table  1.44). 

For  two  outcomes,  the  seven  SACD  programs  combined  had  more  detrimental  impacts  on  growth  for 
children  with  higher  initial  risk  levels.  The  SACD  programs  resulted  in  less  growth  for  child-reported 
Empathy  among  children  with  greater  levels  of  family  risk  (interaction  term  = -0.08  on  table  1.43),  and 
greater  growth  on  Normative  Beliefs  About  Aggression  among  children  with  higher  initial  levels  of  teacher- 
reported  behavior  risks  (interaction  term  = 0.01  on  table  1.45). 
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Table  1.42.  Impacts  on  growth  of  child  outcomes,  by  initial  socioeconomic  risk 


Socioeconomic  risk 

(Mean  = .60;  Standard  deviation  = .71) 

Impact  on  growth  at  the  specified  risk  level  (p-value) 


Scale-Report 

One  standard 
deviation  below  mean1 

Mean2 

One  standard 
deviation  above  mean3 

Interaction  term4 
(p-value) 

Social  and  Emotional  Competence  Domain 

Self-Efficacy  for  Peer  Interactions-CR  (+) 

-0.01 

-0.01 

0.00 

0.01 

(0.500) 

(0.594) 

(0.500) 

(0.511) 

Normative  Beliefs  About  Aggression-CR  (-) 

0.03* 

0.03* 

0.03A 

0.00 

(0.049) 

(0.027) 

(0.079) 

(0.842) 

Empathy-CR  (+) 

-0.01 

-0.02 

-0.03A 

-0.01 

(0.500) 

(0.122) 

(0.059) 

(0.518) 

Behavior  Domain 

Altruistic  Behavior-CR  (+) 

-0.02 

-0.01 

0.01 

0.02 

(0.500) 

(0.628) 

(0.500) 

(0.341) 

Altruistic  Behavior-PCR  (+) 

-0.01 

0.01 

0.04* 

0.03* 

(0.500) 

(0.344) 

(0.045) 

(0.044) 

Altruistic  Behavior-TRS  (+) 

-0.03 

-0.03 

-0.03 

0.00 

(0.500) 

(0.269) 

(0.214) 

(0.870) 

Positive  Social  Behavior-PCR  (+) 

0.01 

0.01 

0.02A 

0.01 

(0.500) 

(0.100) 

(0.090) 

(0.314) 

Positive  Social  Behavior-TRS  (+) 

-0.01 

-0.01 

-0.01 

0.00 

(0.500) 

(0.599) 

(0.500) 

(0.939) 

Problem  Behavior-CR  (-) 

0.01 

0.02 

0.02 

0.01 

(0.500) 

(0.277) 

(0.272) 

(0.677) 

Problem  Behavior-PCR  (-) 

-0.01 

0.00 

0.00 

0.01 

(0.261) 

(0.720) 

(0.500) 

(0.324) 

Problem  Behavior-TRS  (-) 

0.00 

0.00 

0.00 

0.00 

(0.500) 

(0.915) 

(0.500) 

(0.879) 

ADHD-Related  Behavior-TRS  (-) 

0.00 

-0.01 

-0.02 

-0.02 

(0.500) 

(0.716) 

(0.500) 

(0.329) 

See  notes  at  end  of  table. 
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Table  1.42.  Impacts  on  growth  of  child  outcomes,  by  initial  socioeconomic  risk— Continued 


Socioeconomic  risk 
(Mean  = .60;  Standard  deviation  = .71) 


Impact  on  growth  at  the  specified  risk  level  (p-value) 


Scale-Report 

One  standard 
deviation  below  mean1 

Mean2 

One  standard 
deviation  above  mean3 

Interaction  term4 
(p-value) 

Academics  Domain 

Engagement  with  Learning-CR  (+) 

-0.02 

-0.01 

0.01 

0.02 

(0.213) 

(0.644) 

(0.500) 

(0.249) 

Academic  Competence  and  Motivation-TRS  (+) 

-0.01 

0.00 

0.01 

0.01 

(0.500) 

(0.861) 

(0.500) 

(0.559) 

Perceptions  of  School  Climate  Domain 

Positive  School  Orientation-CR  (+) 

-0.02 

-0.03 

-0.04A 

-0.01 

(0.500) 

(0.150) 

(0.087) 

(0.518) 

Negative  School  Orientation-CR  (-) 

0.01 

0.00 

-0.01 

-0.01 

(0.500) 

(0.876) 

(0.500) 

(0.414) 

Student  Afraid  at  School-CR  (-) 

-0.02 

0.00 

0.01 

0.02 

(0.500) 

(0.833) 

(0.500) 

(0.315) 

Victimization  at  School-CR  (-) 

-0.01 

-0.01 

-0.01 

0.00 

(0.500) 

(0.506) 

(0.500) 

(0.965) 

* Significantly  different  from  zero  at  the  .05  level. 

A Significantly  different  from  zero  at  the  .10  to  > .05  level. 

1 The  difference  between  the  estimates  of  the  treatment  and  control  groups'  rate  of  growth,  for  the  average  student  who  is  one  standard  deviation  below  the  mean  for  the  risk 
measure. 

2 The  difference  between  the  estimates  of  the  treatment  and  control  groups’  rate  of  growth,  for  the  average  student  who  is  at  the  mean  for  the  risk  measure. 

3 The  difference  between  the  estimates  of  the  treatment  and  control  groups'  rate  of  growth,  for  the  average  student  who  is  one  standard  deviation  above  the  mean  for  the  risk 
measure. 

4 Estimated  difference  between  the  treatment  and  control  groups  in  the  incremental  change  in  the  rate  of  growth,  with  each  unit  increase  in  the  risk  measure. 

NOTE:  Abbreviations  are 

CR:  Child  Report 

PCR:  Primary  Caregiver  Report 

TRS:  Teacher  Report  on  Student 

ADHD:  Attention  deficit  hyperactivity  disorder 

The  +/-  signs  in  parentheses  indicate  the  direction  of  a beneficial  outcome.  All  impact  estimates  were  calculated  using  HLM  6.06.  Sample  weights  were  used  in  all  analyses  to  (1)  give 
each  program  equal  weight  within  each  time  period,  (2)  give  each  school  equal  weight  in  each  program  (within  each  time  period),  and  (3)  give  each  time  period  equal  weight  in  the 
analysis. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Table  1.43.  Impacts  on  growth  of  child  outcomes,  by  initial  family  risk 


Family  risk 

(Mean  = 1.15;  Standard  deviation  = .18) 

Scale-Report 

Impact  on  growth 

One  standard 
deviation  below  mean1 

at  the  specified  risk  level  (p-value) 

One  standard 
Mean2  deviation  above  mean3 

Interaction  term4 
(p-value) 

Social  and  Emotional  Competence  Domain 

Self-Efficacy  for  Peer  Interactions-CR  (+) 

-0.01 

0.00 

0.01 

0.06 

(0.500) 

(0.739) 

(0.500) 

(0.273) 

Normative  Beliefs  About  Aggression-CR  (-) 

0.02 

0.03* 

0.04A 

0.04 

(0.142) 

(0.047) 

(0.052) 

(0.510) 

Empathy-CR  (+) 

0.00 

-0.02 

-0.03* 

-0.08* 

(0.500) 

(0.115) 

(0.014) 

(0.040) 

Behavior  Domain 

Altruistic  Behavior-CR  (+) 

0.01 

-0.01 

-0.02 

-0.07 

(0.500) 

(0.715) 

(0.300) 

(0.227) 

Altruistic  Behavior-PCR  (+) 

0.00 

0.01 

0.01 

0.02 

(0.500) 

(0.672) 

(0.500) 

(0.799) 

Altruistic  Behavior-TRS  (+) 

-0.05A 

-0.04 

-0.02 

0.07 

(0.070) 

(0.225) 

(0.500) 

(0.129) 

Positive  Social  Behavior-PCR  (+) 

0.01 

0.02A 

0.02* 

0.05A 

(0.500) 

(0.090) 

(0.018) 

(0.090) 

Positive  Social  Behavior-TRS  (+) 

-0.02 

-0.01 

-0.01 

0.04 

(0.500) 

(0.607) 

(0.500) 

(0.443) 

Problem  Behavior-CR  (-) 

0.03A 

0.02 

0.01 

-0.05 

(0.078) 

(0.241) 

(0.500) 

(0.983) 

Problem  Behavior-PCR  (-) 

0.00 

0.00 

0.00 

0.00 

(0.500) 

(0.440) 

(0.500) 

(0.983) 

Problem  Behavior-TRS  (-) 

0.00 

0.00 

0.00 

0.01 

(0.500) 

(0.858) 

(0.500) 

(0.891 

ADHD-Related  Behavior-TRS  (-) 

0.00 

-0.01 

-0.01 

-0.02 

(0.500) 

(0.659) 

(0.500) 

(0.685) 

See  notes  at  end  of  table. 
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Table  1.43.  Impacts  on  growth  of  child  outcomes,  by  initial  family  risk— Continued 


Family  risk 

(Mean  = 1.15;  Standard  deviation 

= .18) 

Impact  on  growth  at  the  specified  risk  level  (p-value) 

One  standard 

One  standard 

Interaction  term4 

Scale-Report 

deviation  below  mean1 

Mean2 

deviation  above  mean3 

(p-value) 

Academics  Domain 

Engagement  with  Learning-CR  (+) 

0.00 

-0.01 

-0.01 

-0.02 

(0.500) 

(0.699) 

(0.500) 

(0.669) 

Academic  Competence  and  Motivation-TRS  (+) 

-0.03 

0.00 

0.02 

0.13* 

(0.222) 

(0.844) 

(0.500) 

(0.031) 

Perceptions  of  School  Climate  Domain 

Positive  School  Orientation-CR  (+) 

-0.01 

-0.03 

-0.04* 

-0.09A 

(0.500) 

(0.170) 

(0.028) 

(0.097) 

Negative  School  Orientation-CR  (-) 

0.00 

0.00 

0.00 

0.00 

(0.500) 

(0.898) 

(0.500) 

(0.926) 

Student  Afraid  at  School-CR  (-) 

-0.03 

-0.01 

0.01 

0.11 

(0.257) 

(0.698) 

(0.500) 

(0.206) 

Victimization  at  School-CR  (-) 

0.01 

-0.01 

-0.04A 

-0.13* 

(0.500) 

(0.466) 

(0.059) 

(0.032) 

* Significantly  different  from  zero  at  the  .05  level. 

A Significantly  different  from  zero  at  the  .10  to  > .05  level. 

1 The  difference  between  the  estimates  of  the  treatment  and  control  groups'  rate  of  growth,  for  the  average  student  who  is  one  standard  deviation  below  the  mean  for  the  risk 
measure. 

2 The  difference  between  the  estimates  of  the  treatment  and  control  groups’  rate  of  growth,  for  the  average  student  who  is  at  the  mean  for  the  risk  measure. 

3 The  difference  between  the  estimates  of  the  treatment  and  control  groups'  rate  of  growth,  for  the  average  student  who  is  one  standard  deviation  above  the  mean  for  the  risk 
measure. 

4 Estimated  difference  between  the  treatment  and  control  groups  in  the  incremental  change  in  the  rate  of  growth,  with  each  unit  increase  in  the  risk  measure. 

NOTE:  Abbreviations  are 

CR:  Child  Report 

PCR:  Primary  Caregiver  Report 

TRS:  Teacher  Report  on  Student 

ADHD:  Attention  deficit  hyperactivity  disorder 

The  +/-  signs  in  parentheses  indicate  the  direction  of  a beneficial  outcome.  All  impact  estimates  were  calculated  using  HLM  6.06.  Sample  weights  were  used  in  all  analyses  to  (1)  give 
each  program  equal  weight  within  each  time  period,  (2)  give  each  school  equal  weight  in  each  program  (within  each  time  period),  and  (3)  give  each  time  period  equal  weight  in  the 
analysis. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Table  1.44.  Impacts  on  growth  of  child  outcomes,  by  initial  perceptions  of  community  risk 


Community  risk 

(Mean  = 1 .52;  Standard  deviation 

i = .61) 

Scale-Report 

Impact  on  growth 

One  standard 
deviation  below  mean1 

at  the  specified  risk  level  (p-value) 

One  standard 
Mean2  deviation  above  mean3 

Interaction  term4 
(p-value) 

Social  and  Emotional  Competence  Domain 

Self-Efficacy  for  Peer  Interactions-CR  (+) 

-0.01 

0.00 

0.01 

0.01 

(0.500) 

(0.829) 

(0.500) 

(0.494) 

Normative  Beliefs  About  Aggression-CR  (-) 

0.04* 

0.03* 

0.02 

-0.02 

(0.024) 

(0.028) 

(0.226) 

(0.384) 

Empathy-CR  (+) 

-0.02 

-0.02A 

-0.02 

0.00 

(0.200) 

(0.096) 

(0.145) 

(0.925) 

Behavior  Domain 

Altruistic  Behavior-CR  (+) 

-0.01 

-0.01 

0.00 

0.01 

(0.500) 

(0.743) 

(0.500) 

(0.669) 

Altruistic  Behavior-PCR  (+) 

0.01 

0.01 

0.00 

-0.01 

(0.500) 

(0.666) 

(0.500) 

(0.665) 

Altruistic  Behavior-TRS  (+) 

-0.05 

-0.03 

-0.02 

0.03 

(0.129) 

(0.258) 

(0.500) 

(0.218) 

Positive  Social  Behavior-PCR  (+) 

0.02 

0.02A 

0.02 

0.00 

(0.146) 

(0.075) 

(0.208) 

(0.939) 

Positive  Social  Behavior-TRS  (+) 

-0.03 

-0.01 

0.00 

0.02 

(0.286) 

(0.615) 

(0.500) 

(0.230) 

Problem  Behavior-CR  (-) 

0.01 

0.02 

0.02 

0.00 

(0.500) 

(0.255) 

(0.307) 

(0.871) 

Problem  Behavior-PCR  (-) 

-0.01* 

0.00 

0.01 

0.02A 

(0.035) 

(0.908) 

(0.250) 

(0.051) 

Problem  Behavior-TRS  (-) 

0.01 

0.00 

-0.01 

-0.02 

(0.500) 

(0.899) 

(0.500) 

(0.222) 

ADHD-Related  Behavior-TRS  (-) 

0.02 

-0.01 

-0.03 

-0.04A 

(0.500) 

(0.728) 

(0.195) 

(0.060) 

See  notes  at  end  of  table. 
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Table  1.44.  Impacts  on  growth  of  child  outcomes,  by  initial  perceptions  of  community  risk — Continued 


4^ 

o 


Community  risk 

(Mean  = 1.52;  Standard  deviation  = .61) 

Impact  on  growth  at  the  specified  risk  level  (p-value) 

One  standard  One  standard  Interaction  term4 

Scale-Report deviation  below  mean1 Mean2  deviation  above  mean3 (p-value) 

Academics  Domain 


Engagement  with  Learning-CR  (+) 

-0.01 

-0.01 

0.00 

0.00 

(0.500) 

(0.661) 

(0.500) 

(0.866) 

Academic  Competence  and  Motivation-TRS  (+) 

0.00 

0.00 

-0.01 

-0.01 

(0.500) 

(0.968) 

(0.500) 

(0.732) 

Perceptions  of  School  Climate  Domain 

Positive  School  Orientation-CR  (+) 

-0.06* 

-0.03A 

-0.01 

0.04- 

(0.017) 

(0.095) 

(0.500) 

(0.073) 

Negative  School  Orientation-CR  (-) 

0.01 

0.00 

0.00 

-0.01 

(0.500) 

(0.792) 

(0.500) 

(0.528) 

Student  Afraid  at  School-CR  (-) 

-0.02 

-0.01 

0.01 

0.02 

(0.500) 

(0.764) 

(0.500) 

(0.510) 

Victimization  at  School-CR  (-) 

0.02 

-0.01 

-0.05* 

-0.06 

(0.224) 

0.386) 

(0.035) 

(0.014) 

* Significantly  different  from  zero  at  the  .05  level. 

A Significantly  different  from  zero  at  the  .10  to  > .05  level. 

1 The  difference  between  the  estimates  of  the  treatment  and  control  groups'  rate  of  growth,  for  the  average  student  who  is  one  standard  deviation  below  the  mean  for  the  risk 
measure. 

2 The  difference  between  the  estimates  of  the  treatment  and  control  groups’  rate  of  growth,  for  the  average  student  who  is  at  the  mean  for  the  risk  measure. 

3 The  difference  between  the  estimates  of  the  treatment  and  control  groups'  rate  of  growth,  for  the  average  student  who  is  one  standard  deviation  above  the  mean  for  the  risk 
measure. 

4 Estimated  difference  between  the  treatment  and  control  groups  in  the  incremental  change  in  the  rate  of  growth,  with  each  unit  increase  in  the  risk  measure. 

NOTE:  Abbreviations  are 

CR:  Child  Report 

PCR:  Primary  Caregiver  Report 

TRS:  Teacher  Report  on  Student 

ADHD:  Attention  deficit  hyperactivity  disorder 

The  +/-  signs  in  parentheses  indicate  the  direction  of  a beneficial  outcome.  All  impact  estimates  were  calculated  using  HLM  6.06.  Sample  weights  were  used  in  all  analyses  to  (1)  give 
each  program  equal  weight  within  each  time  period,  (2)  give  each  school  equal  weight  in  each  program  (within  each  time  period),  and  (3)  give  each  time  period  equal  weight  in  the 
analysis. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Table  1.45.  Impacts  on  growth  of  child  outcomes,  by  child  behavior  risk  as  reported  by  teacher 


Teacher-reported  child  behavior  risk 
(Mean  = 1 .79;  Standard  deviation  = 2.04) 


Scale-Report 

Impact  on  growth  at  the  specified  risk  level  (p-value) 

One  standard  One  standard 

deviation  below  mean1  Mean2  deviation  above  mean3 

Interaction  term4 
(p-value) 

Social  and  Emotional  Competence  Domain 

Self-Efficacy  for  Peer  Interactions-CR  (+) 

0.00 

0.00 

-0.01 

0.00 

(0.500) 

(0.699) 

(0.500) 

(0.929) 

Normative  Beliefs  About  Aggression-CR  (-) 

0.01 

0.03* 

0.05* 

0.01* 

(0.500) 

(0.038) 

(0.005) 

(0.047) 

Empathy-CR  (+) 

-0.01 

-0.02A 

-0.02A 

0.00 

(0.500) 

(0.098) 

(0.064) 

(0.585) 

Behavior  Domain 

Altruistic  Behavior-CR  (+) 

-0.01 

-0.01 

0.00 

0.00 

(0.500) 

(0.786) 

(0.500) 

(0.908) 

Altruistic  Behavior-PCR  (+) 

0.02 

0.01 

-0.01 

-0.01 

(0.188) 

(0.644) 

(0.500) 

(0.228) 

Altruistic  Behavior-TRS  (+) 

-0.05A 

-0.03 

-0.02 

0.01 

(0.091) 

(0.241) 

(0.500) 

(0.142) 

Positive  Social  Behavior-PCR  (+) 

0.00 

0.02A 

0.03* 

0.01A 

(0.500) 

(0.057) 

(0.017) 

(0.070) 

Positive  Social  Behavior-TRS  (+) 

-0.01 

-0.02 

-0.02 

0.00 

(0.500) 

(0.539) 

(0.500) 

(0.755) 

Problem  Behavior-CR  (-) 

0.01 

0.02 

0.02 

0.00 

(0.500) 

(0.248) 

(0.237) 

(0.584) 

Problem  Behavior-PCR  (-) 

-0.01 

0.00 

0.00 

0.00 

(0.307) 

(0.541) 

(0.500) 

(0.431) 

Problem  Behavior-TRS  (-) 

0.01 

0.01 

0.00 

0.00 

(0.500) 

(0.745) 

(0.500) 

(0.563) 

ADHD-Related  Behavior-TRS  (-) 

0.01 

-0.01 

-0.02 

-0.01 

(0.500) 

(0.760) 

(0.500) 

(0.173) 

See  notes  at  end  of  table. 
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Table  1.45.  Impacts  on  growth  of  child  outcomes,  by  child  behavior  risk  as  reported  by  teacher— Continued 


Teacher-reported  child  behavior  risk 

(Mean  = 1 .79;  Standard  deviation  = 2.04) 

Impact  on  growth  at  the  specified  risk  level  (p-value) 


Scale-Report 

One  standard 
deviation  below  mean1 

Mean2 

One  standard 
deviation  above  mean3 

Interaction  term4 
(p-value) 

Academics  Domain 

Engagement  with  Learning-CR  (+) 

0.01 

-0.01 

-0.03 

-0.01 

(0.500) 

(0.544) 

(0.159) 

(0.114) 

Academic  Competence  and  Motivation-TRS  (+) 

-0.03 

-0.01 

0.02 

0.0T 

(0.215) 

(0.799) 

(0.500) 

(0.092) 

Perceptions  of  School  Climate  Domain 

Positive  School  Orientation-CR  (+) 

-0.02 

-0.03 

-0.05* 

-0.01 

(0.500) 

(0.108) 

(0.033) 

(0.257) 

Negative  School  Orientation-CR  (-) 

-0.01 

0.00 

0.02 

0.01 

(0.500) 

(0.762) 

(0.500) 

(0.208) 

Student  Afraid  at  School-CR  (-) 

0.00 

-0.01 

-0.01 

0.00 

(0.500) 

(0.739) 

(0.500) 

(0.896) 

Victimization  at  School-CR  (-) 

-0.01 

-0.01 

-0.01 

0.00 

(0.500) 

(0.518) 

(0.500) 

(0.795) 

* Significantly  different  from  zero  at  the  .05  level. 

A Significantly  different  from  zero  at  the  .10  to  > .05  level. 

1 The  difference  between  the  estimates  of  the  treatment  and  control  groups'  rate  of  growth,  for  the  average  student  who  is  one  standard  deviation  below  the  mean  for  the  risk 
measure. 

2 The  difference  between  the  estimates  of  the  treatment  and  control  groups'  rate  of  growth,  for  the  average  student  who  is  at  the  mean  for  the  risk  measure. 

3 The  difference  between  the  estimates  of  the  treatment  and  control  groups’  rate  of  growth,  for  the  average  student  who  is  one  standard  deviation  above  the  mean  for  the  risk 
measure. 

4 Estimated  difference  between  the  treatment  and  control  groups  in  the  incremental  change  in  the  rate  of  growth,  with  each  unit  increase  in  the  risk  measure. 

NOTE:  Abbreviations  are 

CR:  Child  Report 

PCR:  Primary  Caregiver  Report 

TRS:  Teacher  Report  on  Student 

ADHD:  Attention  deficit  hyperactivity  disorder 

The  +/-  signs  in  parentheses  indicate  the  direction  of  a beneficial  outcome.  All  impact  estimates  were  calculated  using  HLM  6.06.  Sample  weights  were  used  in  all  analyses  to  (1)  give 
each  program  equal  weight  within  each  time  period,  (2)  give  each  school  equal  weight  in  each  program  (within  each  time  period),  and  (3)  give  each  time  period  equal  weight  in  the 
analysis. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Table  1.46.  Impacts  on  growth  of  child  outcomes,  by  child  behavior  risk  as  reported  by  primary  caregiver 


Primary  caregiver-reported  child  behavior  risk 
(Mean  = 3.43;  Standard  deviation  = 2.01) 

Scale-Report 

Impact  on  growth  at  the  specified  risk  level  (p-value) 

One  standard  One  standard 

deviation  below  mean1  Mean2  deviation  above  mean3 

Interaction  term4 
(p-value) 

Social  and  Emotional  Competence  Domain 

Self-Efficacy  for  Peer  Interactions-CR  (+) 

-0.01 

0.00 

0.01 

0.00 

(0.500) 

(0.716) 

(0.500) 

(0.929) 

Normative  Beliefs  About  Aggression-CR  (-) 

0.02 

0.03* 

0.04* 

0.00 

(0.200) 

(0.036) 

(0.028) 

(0.416) 

Empathy-CR  (+) 

-0.01 

-0.02 

-0.02A 

0.00 

(0.299) 

(0.114) 

(0.088) 

(0.490) 

Behavior  Domain 

Altruistic  Behavior-CR  (+) 

-0.01 

0.00 

0.00 

0.00 

(0.500) 

(0.803) 

(0.500) 

(0.854) 

Altruistic  Behavior-PCR  (+) 

0.02 

0.01 

0.00 

0.00 

(0.500) 

(0.531) 

(0.500) 

(0.472) 

Altruistic  Behavior-TRS  (+) 

-0.04 

-0.03 

-0.03 

0.00 

(0.169) 

(0.246) 

(0.315) 

(0.319) 

Positive  Social  Behavior-PCR  (+) 

0.01 

0.02A 

0.02* 

0.00 

(0.500) 

(0.088) 

(0.042) 

(0.178) 

Positive  Social  Behavior-TRS  (+) 

-0.01 

-0.01 

-0.02 

0.00 

(0.500) 

(0.577) 

(0.500) 

(0.900) 

Problem  Behavior-CR  (-) 

0.01 

0.02 

0.02 

0.00 

(0.500) 

(0.246) 

(0.164) 

(0.429) 

Problem  Behavior-PCR  (-) 

-0.01* 

-0.01 

0.00 

0.00 

(0.044) 

(0.419) 

(0.500) 

(0.157) 

Problem  Behavior-TRS  (-) 

0.00 

0.00 

0.00 

0.00 

(0.500) 

(0.886) 

(0.500) 

(0.724) 

ADHD-Related  Behavior-TRS  (-) 

0.00 

-0.01 

-0.02 

-0.01 

(0.500) 

(0.620) 

(0.500) 

(0.113) 

See  notes  at  end  of  table. 
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Table  1.46.  Impacts  on  growth  of  child  outcomes,  by  child  behavior  risk  as  reported  by  primary  caregiver— Continued 


Primary  caregiver-reported  child  behavior  risk 

(Mean  = 3.43;  Standard  deviation  = 2.01 ) 

Impact  on  growth  at  the  specified  risk  level  (p-value) 


Scale-Report 

One  standard 
deviation  below  mean1 

Mean2 

One  standard 
deviation  above  mean3 

Interaction  term4 
(p-value) 

Academics  Domain 

Engagement  with  Learning-CR  (+) 

-0.01 

-0.01 

0.00 

0.00 

(0.500) 

(0.646) 

(0.500) 

(0.711) 

Academic  Competence  and  Motivation-TRS  (+) 

-0.01 

0.00 

0.01 

0.01 

(0.500) 

(0.837) 

(0.500) 

(0.332) 

Perceptions  of  School  Climate  Domain 

Positive  School  Orientation-CR  (+) 

-0.01 

-0.03 

-0.05* 

-0.01 

(0.500) 

(0.126) 

(0.039) 

(0.173) 

Negative  School  Orientation-CR  (-) 

0.00 

0.00 

0.00 

0.00 

(0.500) 

(0.902) 

(0.500) 

(0.898) 

Student  Afraid  at  School-CR  (-) 

0.02 

-0.01 

-0.03 

-0.01 

(0.500) 

(0.791) 

(0.287) 

(0.144) 

Victimization  at  School-CR  (-) 

-0.01 

-0.01 

-0.02 

0.00 

(0.500) 

(0.469) 

(0.500) 

(0.751) 

* Significantly  different  from  zero  at  the  .05  level. 

A Significantly  different  from  zero  at  the  .1 0 to  > .05  level. 

1 The  difference  between  the  estimates  of  the  treatment  and  control  groups’  rate  of  growth,  for  the  average  student  who  is  one  standard  deviation  below  the  mean  for  the  risk  measure. 

2 The  difference  between  the  estimates  of  the  treatment  and  control  groups’  rate  of  growth,  for  the  average  student  who  is  at  the  mean  for  the  risk  measure. 

3 The  difference  between  the  estimates  of  the  treatment  and  control  groups'  rate  of  growth,  for  the  average  student  who  is  one  standard  deviation  above  the  mean  for  the  risk  measure. 

4 Estimated  difference  between  the  treatment  and  control  groups  in  the  incremental  change  in  the  rate  of  growth,  with  each  unit  increase  in  the  risk  measure. 

NOTE:  Abbreviations  are 

CR:  Child  Report 

PCR:  Primary  Caregiver  Report 

TRS:  Teacher  Report  on  Student 

ADHD:  Attention  deficit  hyperactivity  disorder 

The  +/-  signs  in  parentheses  indicate  the  direction  of  a beneficial  outcome.  Sample  weights  were  used  in  all  analyses  to  (1)  give  each  program  equal  weight  in  the  calculation  of  pooled 
impact  estimates  and  (2)  give  each  school  equal  weight  in  each  program. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Table  1 .47.  Sample  size  ranges  for  outcome  analyses 


Outcome  analysis 

Group 

Report 

Year  1 

Year  2 

Year  3 

Mini- 
mum of 
range 

Maxi- 
mum of 
range 

Mini- 
mum of 
range 

Maxi- 
mum of 
range 

Mini- 
mum of 
range 

Maxi- 
mum of 
range 

Combined  program 

Treatment 

CR 

2,181 

2,190 

2,143 

2,162 

2,065 

2,081 

PCR 

1,728 

1,760 

1,668 

1,691 

1,452 

1,473 

TRS 

2,205 

2,252 

2,198 

2,247 

2,073 

2,110 

TRCS 

436 

437 

422 

423 

422 

423 

Control 

CR 

1,939 

1,961 

1,869 

1,892 

1,894 

1,903 

PCR 

1,587 

1,611 

1,475 

1,493 

1,363 

1,383 

TRS 

1,950 

2,025 

1,976 

1,989 

1,921 

1,965 

TRCS 

408 

409 

403 

404 

392 

393 

Individual  programs 

ABC 

Treatment 

CR 

245 

257 

234 

235 

242 

245 

PCR 

194 

197 

168 

171 

167 

167 

TRS 

248 

250 

233 

245 

261 

262 

TRCS 

54 

54 

49 

49 

51 

51 

Control 

CR 

292 

297 

308 

311 

302 

307 

PCR 

217 

224 

229 

229 

214 

219 

TRS 

287 

305 

322 

322 

322 

323 

TRCS 

66 

66 

67 

67 

64 

64 

CSP 

Treatment 

CR 

313 

319 

312 

377 

280 

288 

PCR 

239 

245 

243 

246 

211 

211 

TRS 

324 

325 

302 

404 

265 

266 

TRCS 

50 

52 

57 

91 

62 

62 

Control 

CR 

323 

335 

306 

299 

324 

329 

PCR 

279 

280 

257 

209 

231 

231 

TRS 

329 

344 

320 

333 

324 

326 

TRCS 

50 

50 

60 

82 

60 

61 

LBW 

Treatment 

CR 

345 

347 

338 

339 

331 

332 

PCR 

298 

302 

293 

296 

253 

257 

TRS 

380 

382 

358 

366 

355 

358 

TRCS 

79 

79 

68 

69 

75 

75 

Control 

CR 

264 

265 

238 

240 

220 

221 

PCR 

229 

230 

191 

191 

184 

187 

TRS 

273 

284 

250 

252 

240 

242 

TRCS 

62 

62 

47 

47 

50 

50 

PA 

Treatment 

CR 

296 

298 

280 

284 

258 

259 

PCR 

239 

244 

212 

214 

168 

170 

TRS 

275 

304 

289 

302 

242 

262 

TRCS 

49 

50 

43 

43 

39 

39 

Control 

CR 

277 

280 

220 

225 

241 

242 

PCR 

227 

231 

181 

183 

158 

158 

TRS 

280 

302 

249 

251 

215 

250 

TRCS 

45 

45 

39 

39 

38 

38 

See  notes  at  end  of  table. 
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Table  1 .47.  Sample  size  ranges  for  outcome  analyses— Continued 


Outcome  analysis 

Group 

Report 

Year  1 

Year  2 

Year  3 

Mini- 
mum of 
range 

Maxi- 
mum of 
range 

Mini- 
mum of 
range 

Maxi- 
mum of 
range 

Mini- 
mum of 
range 

Maxi- 
mum of 
range 

PATHS 

Treatment 

CR 

216 

219 

220 

222 

225 

227 

PCR 

191 

198 

181 

186 

169 

174 

TRS 

232 

232 

228 

229 

228 

230 

TRCS 

57 

58 

55 

55 

50 

50 

Control 

CR 

219 

220 

212 

212 

202 

204 

PCR 

179 

180 

166 

168 

161 

164 

TRS 

224 

225 

219 

219 

197 

208 

TRCS 

51 

51 

57 

57 

60 

60 

4Rs 

Treatment 

CR 

401 

403 

372 

377 

350 

355 

PCR 

265 

270 

241 

246 

177 

183 

TRS 

387 

403 

399 

404 

360 

361 

TRCS 

87 

88 

90 

91 

87 

88 

Control 

CR 

301 

303 

292 

299 

306 

309 

PCR 

221 

227 

206 

209 

169 

174 

TRS 

303 

306 

307 

320 

321 

322 

TRCS 

85 

85 

82 

82 

75 

75 

SS 

Treatment 

CR 

358 

361 

383 

387 

375 

376 

PCR 

302 

305 

330 

335 

307 

311 

TRS 

357 

361 

381 

383 

357 

376 

TRCS 

58 

58 

59 

60 

58 

58 

Control 

CR 

259 

261 

288 

291 

293 

297 

PCR 

235 

239 

245 

255 

246 

250 

TRS 

254 

263 

292 

293 

296 

298 

TRCS 

49 

50 

51 

52 

45 

45 

Subgroups 

Gender 

Boys 

CR 

1,983 

1,998 

1,913 

1,933 

1,899 

1,915 

PCR 

1,594 

1,662 

1,503 

1,526 

1,325 

1,349 

TRS 

2,010 

2,067 

2,027 

2,048 

1,927 

1,965 

Girls 

CR 

2,135 

2,153 

2,099 

2,121 

2,060 

2,069 

PCR 

1,688 

1,716 

1,626 

1,645 

1,473 

1,490 

TRS 

2,108 

2,174 

2,132 

2,174 

2,046 

2,090 

New  entrants 

Stayers 

CR 

t 

t 

3,090 

3,124 

2,764 

2,781 

PCR 

t 

t 

2,444 

2,470 

1,953 

1,976 

TRS 

t 

t 

3,230 

3,275 

2,809 

2,862 

New 

entrants 

CR 

t 

t 

921 

931 

1,196 

1,203 

PCR 

t 

t 

699 

714 

862 

879 

TRS 

t 

t 

944 

962 

1,185 

1,217 

Initial  socioeconomic  risk 

CR 

3,292 

3,311 

3,649 

3,693 

3,565 

3,584 

PCR 

2,871 

2,916 

3,076 

3,113 

2,737 

2,775 

TRS 

3,312 

3,411 

3,799 

3,843 

3,588 

3,662 

See  notes  at  end  of  table. 
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Table  1 .47.  Sample  size  ranges  for  outcome  analyses— Continued 


Year  1 Year  2 Year  3 


Mini- 

Maxi- 

Mini- 

Maxi- 

Mini- 

Maxi- 

mum of 

mum  of 

mum  of 

mum  of 

mum  of 

mum  of 

Outcome  analysis 

Group 

Report 

range 

range 

range 

range 

range 

range 

Initial  family  risk 

CR 

3,373 

3,392 

2,749 

2,780 

2,459 

2,473 

PCR 

2,893 

2,939 

2,297 

2,322 

1,834 

1,854 

TRS 

3,382 

3,483 

2,873 

2,901 

2,497 

2,547 

Initial  community  risk 

CR 

3,289 

3,309 

2,702 

2,734 

2,413 

2,427 

PCR 

2,839 

2,884 

2,266 

2,290 

1,813 

1,830 

TRS 

3,298 

3,396 

2,820 

2,857 

2,450 

2,499 

Initial  child  behavior 

risk  (Teacher  Report) 

CR 

3,608 

3,630 

2,936 

2,970 

2,631 

2,647 

PCR 

2,940 

2,988 

2,341 

2,364 

1,870 

1,893 

TRS 

3,567 

3,763 

3,072 

3,111 

2,679 

2,729 

Initial  child  behavior 

risk  (Primary 
Caregiver  Report) 

CR 

3,353 

3,372 

2,732 

2,764 

2,451 

2,464 

PCR 

2,876 

2,923 

2,289 

2,313 

1,828 

1,847 

TRS 

3,365 

3,466 

2,858 

2,886 

2,487 

2,537 

Fidelity  of 
implementation 

High 

CR 

2,020 

2,038 

1,617 

1,635 

1,224 

1,235 

PCR 

1,675 

1,697 

1,323 

1,340 

908 

926 

TRS 

2,018 

2,096 

1,654 

1,679 

1,207 

1,251 

Mixed 

CR 

t 

t 

848 

851 

1,444 

1,450 

PCR 

t 

t 

677 

684 

1,036 

1,052 

TRS 

t 

t 

893 

904 

1,469 

1,485 

Low 

CR 

2,100 

2,113 

1,544 

1,565 

1,289 

1,298 

PCR 

1,640 

1,675 

1,140 

1,158 

870 

877 

TRS 

2,137 

2,182 

1,624 

1,652 

1,317 

1,338 

f Not  applicable. 


NOTE:  Abbreviations  are 

ABC:  Academic  and  Behavioral  Competencies  Program 
CSP:  Competence  Support  Program 
LBW:  Love  in  a Big  World 
PA:  Positive  Action 

PATHS:  Promoting  Alternative  Thinking  Strategies 

4Rs:  The  4Rs  Program  (Reading,  Writing,  Respect,  and  Resolution) 

SS:  Second  Step 
CR:  Child  Report 
PCR:  Primary  Caregiver  Report 
TRS:  Teacher  Report  on  Student 
TRCS:  Teacher  Report  on  Classroom  and  School 
SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Sensitivity  Analyses 

To  implement  the  growth  curve  analyses,  it  was  necessary  to  make  decisions  about  key  model  parameter 
specifications  and  estimation  methods.  A set  of  sensitivity  tests  using  variations  in  key  model  assumptions 
examined  the  original  growth  curve  findings  and  found  them  robust. 

Time  Metric 

Age  of  student  provides  a different  time  metric  by  which  to  measure  the  growth  trajectory.  The  growth  curve 
analysis  was  redone  using  age  in  years  at  the  start  of  implementation  of  the  program.  The  1 8 impact  estimates 
remained  insignificant  when  using  age  as  the  time  metric. 

Weights 

Weights  were  used  to  represent  each  program  equally  within  each  survey  period,  each  school  equally  within 
each  program,  and  each  survey  period  equally  within  the  analysis.  To  determine  whether  the  weights  affected 
the  outcomes,  an  unweighted  analysis  was  conducted.  The  18  impact  estimates  remained  insignificant  in  the 
unweighted  model. 

Fitted  Impacts 

“Fitted”  impact  estimates  and  significance  levels  were  examined  at  follow-up  points  corresponding  to  the 
point-in-time  analyses  at  6 months,  12  months,  18  months,  and  30  months  after  implementation,  which 
correspond  roughly  to  the  times  of  the  data  collection  efforts.  Based  on  the  fitted  growth  curve  model, 
statistically  significant  impact  estimates  were  found  for  three  outcomes  (these  data  are  not  shown  in  a table).45 
One  was  in  a detrimental  direction  and  two  were  in  a beneficial  direction.  At  12,  18,  and  30  months  after 
implementation,  the  seven  SACD  programs,  together,  lowered  growth  of  child-reported  Altruistic  Behavior 
by  .05  (significant  at  the  .01  level),  0.05  (significant  at  the  .01  level),  and  0.06  (significant  at  the  0.05  level), 
respectively.  At  12  months,  18  months,  and  30  months  after  implementation,  the  SACD  programs  increased 
growth  of  parent-reported  Positive  Social  Behavior  by  0.03,  0.04,  and  0.05,  respectively  (all  significant  at  the 
.05  level).  At  12  months,  18  months,  and  30  months  after  implementation,  the  programs  lowered  growth  of 
child-reported  Victimization  by  0.04,  0.05,  and  0.06,  respectively  (all  significant  at  the  .05  level). 

Fall 2005  Data 

The  second  follow-up  point  in  fall  2005  occurred  shortly  after  the  summer,  which  could  affect  the  outcome 
measures  at  fall  2005  through  “learning  loss”  during  the  summer.  The  growth  curve  model  was  estimated 
using  a time  indicator  that  allows  the  growth  coefficient  to  decline  (or  increase)  following  the  summer.  The 
18  impact  estimates  remained  insignificant  after  accounting  for  summer  “learning  loss.” 

Missing  Outcomes 

An  advantage  of  the  growth  curve  methodology  is  that  it  can  accommodate  all  students  who  have  at  least  one 
round  of  data.  The  information  that  was  available  for  each  student  was  used  in  estimating  the  intercept  and 
slope,  across  all  students.  To  assess  the  robustness  of  study  findings  to  the  assumptions  underlying  this 
approach,  the  model  was  estimated  using  only  those  students  with  (1)  all  rounds  of  follow-up  data,  and 
(2)  the  last  round  of  follow-up  data  in  spring  2007  and  at  least  one  earlier  round.  The  impact  estimates 
remained  insignificant  when  using  these  restricted  samples. 


45  Point-in-time  impacts  were  calculated  by  adding  (1)  the  estimated  impact  of  the  treatment  at  initial  data  collection  to 
(2)  the  estimated  average  annual  impact  of  the  treatment  on  the  growth  curve  trajectory  multiplied  by  the  number  of 
years  since  implementation.  For  the  three  significant  outcomes,  the  estimated  difference  between  the  treatment  and 
control  groups  at  initial  data  collection,  though  insignificant,  was  in  the  same  direction  as  the  estimated  slope.  Thus,  the 
cumulative  impact  of  the  estimated  difference  in  the  slope  between  the  treatment  group  and  control  group,  in  addition 
to  the  estimated  difference  at  initial  data  collection  between  the  treatment  group  and  control  group,  resulted  in 
significant  differences  at  later  time  points.  However,  the  cumulative  impact  of  the  difference  in  the  average  annual  slope 
between  the  treatment  group  and  control  group  (that  is,  the  estimated  difference  in  the  change  in  the  outcomes  of  the 
treatment  group  and  control  group)  resulted  in  no  significant  differences  for  any  of  the  point-in-time  estimates. 
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Discussion 

IES  and  DVP  established  the  SACD  Research  Program  to  rigorously  evaluate  the  effects  of  seven  universal, 
schoolwide  programs  to  improve  students’  social  and  character  development.  The  programs  evaluated  were 
chosen  through  a peer-review  process  of  submitted  applications.  A separate  experiment  using  random 
assignment  of  schools  was  done  for  each  program.  Within  each  program,  participating  schools  were  stratified 
into  similar  pairs  then  randomly  assigned  to  receive  the  treatment  of  the  intervention  or  continue  with 
whatever  SACD  activities  they  were  using.  All  students  in  the  elementary  grades  within  the  treatment  schools 
received  the  intervention  program  over  the  3 years  of  the  study.  For  the  evaluation,  one  cohort  of  students 
was  followed  from  the  beginning  of  third  grade  through  the  end  of  fifth  grade.  A third  party  (Mathematica 
Policy  Research,  Inc.)  provided  technical  assistance  in  the  sampling  design,  helped  design  data  collection, 
conducted  and  coordinated  the  data  collection,  and  designed  and  conducted  the  analyses.  Twenty  outcomes 
were  used  to  measure  impacts  on  students  and  perceptions  of  school  climate  and  these  were  grouped  under 
four  domains:  Social  and  Emotional  Competence,  Behavior,  Academics,  and  Perceptions  of  School  Climate. 
The  data  were  analyzed  from  all  seven  programs  together  and  by  individual  program  to  determine  the 
collective  and  individual  program  impacts. 

Under  such  a design,  inferences  can  be  made  regarding  the  programs,  combined  and  individually,  as  the  cause 
of  any  impacts  on  these  domains  and  the  outcomes  that  define  them.  Rigorous  inferences  can  be  drawn  only 
about  the  specific  programs  tested  in  the  context  (including  the  schools)  in  which  they  were  tested.  Other 
programs  tested  in  similar  or  different  settings  and  these  programs  tested  in  different  settings  might  have 
yielded  different  findings.  Therefore,  inferences  cannot  be  made  beyond  this  set  of  programs  and  contexts. 

An  analysis  of  the  initial  data  found  few  significant  differences  between  the  family,  teacher,  and  school 
characteristics  of  the  treatment  and  control  students,  a sign  that  the  randomization  of  the  schools  had  created 
similar  treatment  and  control  groups  (similar  at  least  on  the  observed  characteristics).  Treatment  teachers 
reported  significantly  greater  use  of  and  training  in  SACD  activities  than  control  teachers  more  often  than 
would  be  expected  by  chance.  Because  the  training  of  all  treatment  teachers  and  the  implementation  of  six  of 
the  programs  occurred  before  the  initial  data  were  collected,  it  could  not  be  determined  whether  these 
differences  resulted  from  actual  baseline  differences  between  treatment  and  control  schools  and  teachers  (i.e., 
that  randomization  did  not  create  similar  treatment  and  control  groups)  or  from  implementation  occurring 
before  the  initial  data  collection.  The  control  group  for  the  SACD  evaluation  was  a “standard  practice” 
control,  not  a “no  treatment”  control,  in  that  control  teachers  reported  the  use  of  SACD  activities  as  well  as 
related  professional  development. 

The  SACD  programs  were  found  to  have  a statistically  significant  impact  on  the  reported  use  of  SACD 
activities  in  the  classroom  and  school.  Treatment  teachers  reported  significantly  greater  implementation  of 
activities  than  control  teachers  for  127  of  the  249  activity  outcomes  measured  over  the  3 years.  In  addition,  in 
each  year  four  of  the  six  SACD  activity  domains  were  found  to  be  significantly  positively  affected  by  the 
SACD  programs.  These  four  domains  included  the  use  of  any  SACD  activities,  the  use  of  SACD  activities 
linked  to  named  programs,  the  use  of  classroom  materials  and  instructional  strategies  when  teaching  SACD 
activities,  and  professional  development  in  using  SACD  activities.  The  actual  differences  in  reported  use  of 
SACD  activities  between  the  treatment  and  control  groups  varied  from  5 to  34  percentage  points.  The  results 
provided  evidence  that  implementing  the  SACD  programs  increased  the  reported  use  of  SACD  activities  in 
the  schools. 

To  examine  the  average  effects  of  the  seven  programs  (combined  and  individually)  on  student  outcomes,  two 
sets  of  analyses  were  done.  The  first  analyzed  the  outcomes  on  a year-by-year  basis  (from  the  fall  of  third 
grade  to  the  spring  of  third  grade,  spring  of  fourth  grade,  and  spring  of  fifth  grade)  for  the  entire  sample,  the 
individual  programs,  and  a set  of  subgroups.  The  second  analyzed  the  impacts  on  the  average  annual  growth 
in  student  outcomes  for  the  overall  sample,  the  individual  programs,  and  the  subgroups.  For  both  analyses, 
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the  ability  to  detect  smaller  significant  effects  was  greater  when  examining  the  programs  all  together  versus 
examining  the  individual  programs  or  the  subgroups  because  of  the  differences  in  sample  sizes. 

Neither  the  year-by-year  analysis  nor  the  growth  curve  analysis  found  that  the  seven  SACD  programs 
improved  student  outcomes  when  considered  together,  individually  by  program,  or  for  specific  subgroups. 
For  the  combined-program  analysis,  the  year-by-year  analysis  found  fewer  significant  impacts  than  expected 
by  chance  (2  out  of  60  estimated  impacts)  and  the  growth  analysis  found  no  significant  impacts.  For  the 
individual  program  analysis,  the  year-by-year  analysis  found  fewer  significant  impacts  than  expected  by  chance 
(16  out  of  420  estimated  impacts)  with  9 having  beneficial  impacts  and  7 having  detrimental  impacts.  The 
growth  analysis  found  the  same  number  of  impacts  as  expected  by  chance  (6  out  of  126  estimated  impacts) 
with  2 having  beneficial  impacts  and  4 having  detrimental  impacts.  For  the  subgroup  analyses,  the  year-by- 
year analysis  found  more  significant  impacts  than  expected  by  chance  for  gender  (8  out  of  54  estimated 
impacts)  and  initial  risk  levels  (41  out  of  270  estimated  impacts)  but  not  for  stayers  versus  new  entrants.  For 
gender,  half  of  the  significant  impacts  showed  a beneficial  impact  of  the  intervention  and  half  showed  a 
detrimental  impact.  For  initial  risk  level,  26  showed  a beneficial  impact  of  the  intervention  on  high-risk 
students  and  15  showed  a detrimental  impact  on  high-risk  students.  The  growth  analysis  found  fewer 
significant  impacts  than  expected  by  chance  for  all  the  subgroups  except  those  based  on  initial  risk  levels,  for 
which  6 out  of  90  estimated  impacts  were  significant.  Four  of  these  6 significant  impacts  were  beneficial  and 
2 were  detrimental.  In  sum,  the  SACD  multiprogram  evaluation  provides  no  evidence  that  the  seven 
universal,  schoolwide  programs  improved  students’  social  and  character  development. 

Several  explanations  for  this  finding  can  be  considered:  (1)  failure  of  the  conceptualization  and  design  of  the 
intervention,  (2)  weak  implementation  of  the  intervention,  (3)  nonsubstantial  differences  in  the  level  of 
SACD  activities  in  the  treatment  and  control  schools,  and  (4)  methodological  limitations  of  the  evaluation. 

Failure  of  the  conceptualization  and  design  of  the  intervention  refers  to  the  possibility  that  the  seven 
programs  tested  might  not  have  altered  students’  social  and  character  development  in  the  expected  ways 
because  the  theories  underlying  them  or  the  combinations  of  activities  chosen  to  bring  about  the  desired 
changes  in  students’  attitudes  and  behaviors  were  inadequate  for  the  purpose.  For  example,  one  alternative 
view  to  that  adopted  in  the  SACD  evaluation  is  that  only  a subset  of  elementary-age  children  has  deficits  in 
social  behavior  and  character,  and  these  deficits  require  a more  targeted,  more  intensive  intervention  than 
schoolwide  programs  can  provide.  Therefore,  if  a school-based  program  is  to  be  effective,  a combination  of 
schoolwide  and  targeted  activities  might  be  required  for  the  intervention  to  make  a significant  difference  in 
students’  outcomes  (e.g.,  see  Conduct  Problems  Prevention  Research  Group  1999).  These  kinds  of 
explanations,  if  confirmed  by  other  studies,  would  lead  to  focusing  more  effort  on  understanding  how  social 
and  character  development  occurs  among  elementary-age  children,  how  this  development  can  be  affected, 
and  what  types  of  practices  in  classrooms  and  schools  can  be  used  to  bring  about  desired  effects. 

Weak  implementation  of  the  intervention  refers  to  the  possibility  that,  although  the  intervention  might  have 
been  well  conceived  and  well  designed,  the  treatment  schools  did  not  implement  these  practices  effectively  on 
average.  Weakly  implemented  programs  may  not  have  positive  impacts  on  students.  The  SACD  evaluation 
examined  seven  different  SACD  programs,  each  with  unique  features,  and  the  fidelity  rating  used  measured 
fidelity  of  implementation  relative  to  the  targets  established  for  each  specific  intervention.  In  this  way,  the 
ratings  were  standardized  relative  to  each  site’s  program-specific  benchmarks  and  could  be  compared  across 
programs  and  years.  The  analysis  of  the  fidelity  data  found  little  evidence  of  a relationship  between  high 
fidelity  and  more  beneficial  outcomes.  The  number  of  significant  associations  found  between  fidelity  and 
beneficial  outcomes  was  higher  than  expected  by  chance  (5  versus  3 out  of  54  estimated  impacts)  but  4 of  the 
5 significant  results  were  due  to  detrimental  associations  between  low  fidelity  and  outcomes  rather  than 
beneficial  associations  between  high  fidelity  and  outcomes.  The  approach  used  to  obtain  comparable  fidelity 
ratings  required  two  compromises.  First,  it  could  not  account  for  differences  among  the  programs’ 
implementation  standards — for  example,  whether  programs  differed  in  how  difficult  they  were  to  implement. 
Second,  it  provided  no  information  on  why  implementation  was  of  a certain  quality.  As  a result,  the  SACD 
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evaluation  fidelity  measure  may  not  provide  adequate  information  regarding  the  issue  of  whether  low  fidelity 
might  have  been  the  reason  behind  the  lack  of  significant  findings.  The  fidelity'  measures  used  by  each 
research  team  using  team-chosen  criteria  may  provide  additional  information  on  how  well  each  program  was 
implemented  in  each  treatment  school. 

Nonsubstantial  differences  in  the  level  of  SACD  activities  refers  to  the  possibility  that  the  implementation 
differences  between  the  treatment  and  control  schools  were  not  great  enough  to  generate  statistically 
significant  differences  in  student  outcomes.  Like  the  treatment  schools,  the  control  schools  joined  the  study 
with  a willingness  to  implement  a SACD  program,  showing  a willingness  to  promote  social  and  character 
development.  In  addition,  some  of  the  sites  were  located  in  states  where  legislation  required  or  promoted 
such  activities.  The  control  group  represented  “standard  practice,”  which  included  the  reported  use  of  SACD 
activities  in  the  classroom.  For  example,  86  percent  to  90  percent  of  control  teachers  reported  using  activities 
to  promote  any  one  of  the  six  SACD  goals.  While  a statistically  significant  larger  percentage  of  teachers  in  the 
treatment  schools  reported  conducting  such  activities  (95%  to  96%)  the  5-  to  1 0-percentage -point  differences 
may  not  have  been  large  enough  to  lead  to  improved  student  outcomes.  At  the  same  time,  the  significant 
differences  between  treatment  and  control  teacher  reports  were  larger  for  other  responses  regarding  the  use 
of  SACD  activities.  For  example,  the  differences  between  treatment  and  control  teachers  regarding  the  use  of 
activities  from  named  programs  were  29  to  34  percentage  points  across  the  3 years.  These  results,  plus  the 
finding  that  treatment  teachers  reported  greater  use  of  some  instructional  materials  and  methods  to  promote 
SACD  goals,  provide  evidence  that  the  treatment  teachers  were  making  a more  intensive  effort  to  promote 
social  and  character  development. 

There  are  three  methodological  limitations  of  the  evaluation  that  may  have  contributed  to  the  finding  of  no 
impacts  on  student  outcomes.  First,  the  evaluation  relied  on  self-reported  data  by  teachers  and  principals 
regarding  the  use  of  SACD  activities.  No  observational  studies  were  done  to  validate  these  reports.  If 
treatment  teachers  over-reported  their  use  of  SACD  activities  (possibly  because  they  felt  an  expectation  to 
report  high  use  given  that  a SACD  program  was  being  implemented  in  their  school),  the  impacts  of  the 
treatment  could  be  misestimated;  that  is,  if  there  really  were  no  differences  in  the  level  of  SACD  activities 
between  the  treatment  and  control  groups  then  a lack  of  effects  might  be  expected.  The  percentages  of 
treatment  teachers  (95%  to  96%)  who  reported  using  any  activities  to  promote  one  of  the  six  SACD  goals 
differed  from  the  percentages  who  reported  using  activities  from  named  programs  (68%  to  72%)  for  the 
same  purpose.  This  difference  does  not  rule  out  the  possibility  of  systematic  over -reporting;  however,  it  does 
suggest  that  some  teachers  were  candid  in  their  reporting  on  their  use  of  the  treatment  programs.  The 
research  teams  used  observations  or  product  measures  to  check  implementation  of  their  specific  programs, 
and  the  results  from  this  work  may  provide  additional  evidence  about  the  potential  importance  of  over- 
reporting of  implementation  by  treatment  teachers. 

A second  methodological  limitation  was  that  student-provided  data  (used  for  12  of  the  20  outcomes)  were 
not  available  for  36  percent  to  39  percent  of  students,  depending  on  the  year,  because  parents  did  not  provide 
written  consent  or  students  did  not  assent  to  take  part  in  the  study.  Primary  caregiver  data,  used  for  three 
outcomes,  were  not  available  for  49  percent  to  54  percent  of  students.  It  is  possible  that  students  included  in 
the  study  differed  from  those  not  included  due  to  an  absence  of  data.  As  the  study  did  not  collect  descriptive 
data  on  the  nonobserved  students,  the  existence  of  such  differences  could  not  be  determined,  and  how  the 
inclusion  of  these  students  in  the  study  would  have  affected  the  findings  is  unknown.  Given  the  few 
statistically  significant  and  substantively  important  impacts  found  with  the  existing  sample,  there  would  need 
to  be  a large  and  consistent  impact  on  the  nonobserved  students  (had  they  been  included)  to  change  the 
findings.  For  example,  because  two-thirds  of  the  population  who  were  observed  received  an  average  impact 
of  zero,  the  nonobserved  one-third  would  need  to  have  received  an  average  impact  of  nearly  one-third  of  a 
standard  deviation  to  bring  the  overall  mean  to  one -tenth  of  a standard  deviation  unit.  In  addition,  because 
the  subgroup  analyses  did  not  find  systematically  significant  impacts,  there  is  no  evidence  that  should  these 
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missing  students  come  from  one  of  the  subgroups  (e.g.,  higher  initial  risk)  their  inclusion  would  change  the 
findings. 

A third  methodological  limitation  was  the  sample  size  for  the  individual  program  evaluations.  The  combined 
analysis  of  all  seven  programs  provided  a sample  size  sufficient  to  detect  student-level  impacts  at  MDES 
ranging  from  0.03  to  0.23  standard  deviations  (with  more  than  75%  of  them  below  0.10  standard  deviations). 
The  power  to  detect  impacts  at  the  level  of  the  individual  programs  was  more  limited  (the  MDES  ranged 
from  0.1  to  0.9  over  the  3 years)  and  individual  program-level  effects  might  have  been  missed.  To  address  this 
limitation  at  the  program  level,  nonsignificant  impacts  of  at  least  0.25  standard  deviations  were  identified  as 
substantively  important  results  and  only  a small  number  were  identified,  with  an  almost  equal  division  into 
beneficial  and  detrimental  effects. 

The  seven  programs  tested  in  the  SACD  evaluation  were  similar  in  being  coherent,  universal,  school-based 
programs  to  promote  social  and  character  development  of  third-  to  fifth-grade  students.  They  were  diverse 
both  in  their  specific  goals  and  in  their  approaches  to  promoting  social  and  character  development  for  all 
students.  In  addition,  they  were  evaluated  in  diverse  types  of  locales  in  which  schools  served  communities 
with  very  different  ethnic  and  socioeconomic  compositions.  On  average,  the  seven  programs  did  not  improve 
students’  social  and  emotional  competence,  behavior,  academic  achievement,  and  student  and  teacher 
perceptions  of  school  climate.  In  addition,  although  the  numbers  of  schools  and  students  in  each  program 
were  not  always  sufficient  to  draw  firm  conclusions  at  the  program  level,  the  patterns  of  estimated  impacts 
for  each  program  were  largely  similar:  students’  outcomes  were  not  affected. 
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University  at  Buffalo,  The  State  University  of  New  York 

(Buffalo  Site) 


Intervention 

Researchers  at  the  University  at  Buffalo,  The  State  University  of  New  York  (SUNY)  (Buffalo  site) 
implemented  and  evaluated  the  Academic  and  behavioral  Cotnpetencies  Program  (ABC)  (Pelham,  Fabiano,  and 
Massetti  2005;  Smith  et  al.  2007;  Waschbusch,  Pelham,  and  Massetti  2005).  ABC  was  developed  at  the  Center 
for  Children  and  Families  at  the  University  at  Buffalo,  SUNY.  This  program  is  designed  to  develop  and 
maintain  a positive  learning  environment  that  reduces  misbehavior  and  promotes  positive  skills  and 
competencies.  It  focuses  on  classroom  behavior  management  and  teaching  and  promoting  social  skills  and 
includes  multiple  components.  Table  2.1  describes  ABC’s  general  characteristics  (panel  1),  the  types  of 
instruction  and  strategies  used  (panel  2),  the  professional  development  provided  for  those  implementing  the 
program  (panel  3),  and  the  social  and  character  development  activities  (panel  4)  and  outcomes  (panel  5) 
addressed  by  the  program. 

• ABC  components  include  schoolwide  rules,  a response/ cost  system  for  tracking  and  responding  to 
disruptive  student  behavior  (e.g.,  behavior  color  charts),  a daily  home  note  or  communication  tool 
for  students  who  follow  the  school  rules,  a weekly  reinforcement  activity  (Fun  Friday),  and  a teacher- 
led  social  skills  program. 

• ABC  program  staff  conduct  a program  development  workshop  with  a school  committee  of 
representative  teachers,  staff,  and  administrators  during  the  summer  prior  to  initial  implementation. 
The  workshop  is  intended  to  discuss  components  of  the  program  and  to  modify  the  program  within 
the  designed  framework  to  meet  the  needs  of  the  school.  The  committee  develops  a set  of 
schoolwide  rules  and  consequences  that  form  the  backbone  of  the  program’s  implementation  in  the 
school. 

• Teachers  and  school  staff  are  trained  in  behavior  management  skills  and  strategies  and  on  delivery  of 
the  ABC  components  and  structure.  Teacher  training  covers  general  classroom  behavior 
management  skills  for  reducing  disruptive  behavior  and  promoting  behavioral  competencies  and 
strategies  specific  to  ABC.  Teachers  are  taught  skills  such  as  building  positive  relationships  with 
students,  using  labeled  praise  as  social  reinforcement,  setting  appropriate  limits,  and  implementing 
effective  discipline  strategies. 

• Consistent  schoolwide  rules  are  posted  and  monitored.  Students  who  follow  the  rules  earn  daily 
notes,  which  are  used  to  earn  weekly  Fun  Friday  activities  and  other  events  and  privileges. 

• Behavioral  consultants  are  a standard  feature  of  the  ABC  program.  They  spend  20  hours  per  week  at 
the  school  and  provide  ongoing  observations  and  consultations  with  teachers  on  the  use  of  behavior 
management  strategies  and  ABC  components.  Consultation  with  teachers  ranges  from  discussing 
general  program  implementation  to  training  on  how  to  make  program  modifications  to  address  the 
needs  of  children  who  could  benefit  from  more  intensive  implementation  of  the  program. 

• Each  day  for  10  minutes,  teachers  implement  scripted  social  skills  training  in  the  classroom.  In  these 
lessons,  they  illustrate  the  targeted  social  skill  that  is  intended  to  enhance  problem  solving, 
communication,  and  cooperation,  and  then  encourage  practice  among  students.  Teachers  have 
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opportunities  to  reinforce  and  promote  these  social  skills  throughout  the  day  using  “caught  being 
good”  tickets  and  prizes. 

• The  behavioral  consultants  at  the  schools  also  deliver  parenting  workshops.  Seven  workshops  are 
offered  that  address  general  parenting  topics  such  as  building  positive  relationships  with  children  and 
using  effective  discipline  strategies. 

• A targeted  recreational/ peer  skills  program  is  delivered  to  schools  for  students  who  exhibit  high  rates 
of  disruptive  behavior  in  the  school  and  classrooms.  The  recreational  program  takes  place  either 
during  the  school  day  or  after  school,  and  it  focuses  on  building  social  competencies,  enhancing  rule- 
following behavior,  and  building  positive  relationships  with  peers. 

• Individualized  programs,  typically  in  the  form  of  modifications  of  the  classwide  program  (e.g.,  more 
frequent  feedback,  changes  in  goal  criteria  or  rewards)  and  daily  report  cards,  are  developed  and 
implemented  through  consulting  with  teachers  about  students  who  have  high  rates  of  disruptive 
behaviors. 
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Table  2.1 . Academic  and  Behavioral  Competencies  Program 

Panel  1 : General  characteristics 

Target  population 

Universal  and  targeted 

Program  components 

Peer:  In  and  out  of  class 

Parent  Daily  positive  notes  and  monthly  parenting  workshops 
Classroom:  Teacher  training  and  coaching,  behavior  management,  and  lessons 
Schoolwide:  Planned  events,  reward  programs,  schoolwide  rules,  and  discipline  policies 
Community:  None  or  not  major  focus 

Training:  Pretraining  and  ongoing  behavior  consultation  through  school-based  consultant 
Level  of  integration 

Add-on  curriculum  and  classroom  and  schoolwide  activities 
Flexibility 

Manualized:  Manual  includes  modules  for  all  program  components  (training,  universal,  and  targeted  levels); 

manual  adapted  for  each  school  with  input  from  school  committees 
Adaptability:  Program  may  be  individualized  within  the  framework  at  the  school,  classroom,  and  individual  child 

level 

See  notes  at  end  of  table. 


Panel  2:  Description  of  instruction  and  strategies 

Classroom 

Lessons 

Who  delivers:  T eacher 

Activities  and  tools:  Social  skill  of  the  day  description  and  classroom  or  schoolwide  role-play 
Content:  Social  skills,  problem  solving,  classroom  management,  and  peer  relationships 
Frequency:  Social  skills  training:  daily  5-  to  10-minute  lessons;  reinforcement  and  application:  daily 
throughout  the  day;  recreational  peer  program:  45  minutes  two  times  per  week 
Strategies 

Who  delivers:  Teacher  (universal);  consultants  and/or  counselors  (targeted) 

Activities  and  tools:  Schoolwide  rules;  social  reinforcement  for  positive  behavior;  classroom  management 
procedures;  teacher  training  in  classroom  management,  effective  discipline,  and  building  teacher- 
child  relationships;  rewards  for  rule-following  and  consequences  for  negative  behavior  (e.g.,  time  out) 
Frequency:  Daily 

Supplement  to  classroom 

Behavioral  consultant  services  to  teachers/classrooms;  peer  mediation;  discipline/office  referral  policies  and 
procedures;  parenting  workshops;  targeted  peer  program  for  disruptive  children 

Schoolwide  activities 

Schoolwide  rules;  classroom  management  procedures;  behavior  honor  roll;  schoolwide  recognition  and 

incentives;  schoolwide  discipline  policies 

See  notes  at  end  of  table. 
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Table  2.1 . Academic  and  Behavioral  Competencies  Program — Continued 

Panel  3:  Professional  development 

Pre-implementation 

Teachers 

Content:  Teacher  training  in  classroom  behavioral  management  concepts  and  strategies;  development  of 
schoolwide  rules,  discipline  policies  and  procedures,  and  classroom  management  policies  with  school 
committee 

Duration:  3 half-days  (9  hours  total) 

Other 

Content:  Staff,  same  as  teacher  training 
Duration:  3 half-days  (9  hours  total) 

Ongoing  consultation 
Teachers 

Content:  Teacher  consultation;  coaching  on  use  of  classroom  management;  school  consultation  on  policies 
and  procedures 

Duration:  Monthly  (at  minimum)  or  more  frequently  per  teacher  or  school  request 
Other 

Content:  Consultation  on  school  discipline  policies  and  procedures,  such  as  lunchroom  discipline,  bus 
discipline,  hallway  policies,  etc. 

Duration:  As  needed,  with  regular  follow-up 

See  notes  at  end  of  table. 


Panel  4:  Activities  for  SACD  goals 


Violence  prevention  and  peace  promotion 
Social  and  emotional  development 
Character  education 


✓ 

✓ 


Risk  prevention  and  health  promotion 
Civic  responsibility  and  community  service 
Behavior  management 


✓ 


Tolerance  and  diversity 
See  notes  at  end  of  table. 


Panel  5:  SACD  outcomes  addressed 


Engagement  with  Learning 
Academic  Competence  and  Motivation 
Altruistic  Behavior 
Positive  Social  Behavior 
Problem  Behavior 


✓ 

✓ 


Empathy 

Positive  School  Orientation 
Negative  School  Orientation 
Student  Afraid  at  School 
Victimization  at  School 


✓ 
✓ 

NOTE:  Abbreviations  are 

v : Activity  or  outcome  addressed 
Blank  cell:  Activity  or  outcome  not  addressed 
SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 


Self-Efficacy  for  Peer  Interactions 
Normative  Beliefs  About  Aggression 


Feelings  of  Safety 
Student  Support  for  Teachers 
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Sample  and  Random  Assignment 

The  Buffalo  research  team  recruited  a total  of  10  public  and  2 charter  elementary  schools.  The  12  schools 
were  in  3 school  districts  in  upstate  New  York.  The  schools  were  randomly  assigned  to  treatment  and  control 
conditions  prior  to  the  fall  2004  initial  data  collection.46  A two-step  process  was  used.  First,  a computer- 
generated pairwise  matching  algorithm  developed  by  Mathematica  Policy  Research,  Inc.  (MPR)  identified  the 
best  pairwise  matches  across  the  12  schools  based  on  variables  identified  by  the  Buffalo  research  team.  The 
variables  included  (a)  school  district,  (b)  percentage  of  students  eligible  for  free  or  reduced-price  lunch  during 
the  2003-04  school  year,  (c)  total  number  of  students  enrolled  during  the  2003-04  school  year,  (d)  percentage 
of  minority  children  enrolled  during  the  2003-04  school  year,  and  (e)  percentage  of  students  achieving 
mastery  level  or  better  scores  on  New  York’s  fourth-grade  English  language  arts  standardized  achievement 
test  administered  during  the  2002-03  school  year.  Second,  using  a random  numbers  table,  1 school  in  each 
matched  pair  was  assigned  to  either  the  intervention  or  the  control  condition,  resulting  in  6 schools  receiving 
the  ABC  program  and  6 schools  acting  as  control  schools.  Control  schools  in  this  study  were  a “business  as 
usual”  comparison,  meaning  these  schools  implemented  the  social  and  character  development  activities  that 
constituted  their  standard  practice.  The  level  of  social  and  character  development  practices,  including 
activities,  materials,  teacher  strategies,  and  professional  development  activities,  in  both  the  treatment  and 
control  schools  was  documented  in  order  to  understand  the  nature  of  the  comparison  being  made  between 
the  intervention  schools  and  the  standard  practice  control  schools.  Assignment  to  treatment  or  control 
condition  at  the  school  level  limited  the  risk  of  contamination  between  treatment  and  control  classrooms.  All 
12  schools  remained  in  the  study  for  the  full  3 years.  One  school  converted  to  a K-8  school  but  this  did  not 
affect  the  students  in  the  study. 

The  original  student  sample  (the  cohort  of  students  in  the  third  grade  in  the  12  schools  in  fall  2004) 
numbered  879  students  (380  treatment  and  499  control).  Table  2.2  documents  the  change  in  the  sample  over 
the  three  spring  follow-up  data  collection  periods.  Over  time,  new  entrants  to  the  cohort  became  a larger 
percentage  of  the  sample,  eventually  making  up  32  percent  of  the  sample  by  the  spring  of  Year  3.  Only  in 
Year  1 was  there  a statistically  significant  difference  between  the  treatment  and  control  groups  in  the  number 
of  new  entrants  (in  this  case,  fewer  new  entrants  in  the  treatment  group).  The  percentage  of  the  sample  made 
up  of  the  original  cohort  further  declined  as  students  left  the  schools.  By  Year  3,  approximately  33  percent  of 
the  original  sample  had  left.  There  were  no  significant  differences  between  treatment  and  control  leavers. 


46  In  Year  2 (fall  2005),  the  Buffalo  research  team  recruited  two  more  schools,  one  assigned  to  treatment  and  one 
assigned  to  control,  which  were  followed  to  the  end  of  the  study  (spring  2007).  A description  of  this  second  cohort  and 
all  relevant  findings  can  be  found  in  appendix  A.  The  data  from  this  second  cohort  are  not  included  in  the  analyses  and 
results  reported  in  this  chapter. 
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Year  1 Year  1 Year  2 Year  3 

(Fall  3rd  grade) (Spring  3rd  grade)  (Spring  4th  grade)  (Spring  5th  grade) 


Treat-  Treat-  Treat-  Treat- 


Characteristic 

Total 

ment 

Control 

Total 

ment 

Control 

Total 

ment 

Control 

Total 

ment 

Control 

School  sample  size 

12 

6 

6 

12 

6 

6 

12 

6 

6 

12 

6 

6 

Student  sample  size 

879 

380 

499 

875 

373 

502 

877 

367 

510 

871 

353 

518 

Stayers 

t 

t 

T 

836 

363 

473 

719 

308 

411 

590 

245 

345 

New  entrants 

t 

t 

t 

39 

10 

29 

158 

59 

99 

281 

108 

173 

New  entrants  as  a percent  of 
spring  enrollment 

t 

t 

t 

4.5 

2.7* 

5.8 

18.0 

16.1 

19.4 

32.3 

30.6 

33.4 

Total  leavers  (from  original  cohort) 

t 

T 

t 

43 

17 

26 

160 

72 

88 

289 

135 

154 

Leavers  as  a percent  of  fall  2004 
enrollment 

t 

T 

t 

4.9 

4.5 

5.2 

18.2 

18.9 

17.6 

32.9 

35.5 

30.9 

Number  of  students  per  school 

(mean) 

73 

63 

83 

73 

62 

84 

74 

62 

85 

73 

59 

86 

Range  of  number  of  students  per 

school 

48-115 

48-84 

64-115 

47-118 

47-83 

66-118 

41-131 

41-80 

63-131 

45-125 

45-77 

49-125 

t Not  applicable. 

* Treatment  group  significantly  different  from  control  group  at  the  .05  level. 
SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Implementation 

Training 

The  intervention  teachers  received  9 hours  of  program  implementation  training  prior  to  the  beginning  of  the 
school  year  (see  panel  3 of  table  2.1).  If  teachers  were  unable  to  attend  this  initial  training  due  to  scheduling 
conflicts,  they  were  provided  with  make-up  training  by  the  program  staff  and  behavioral  consultants. 
Teachers  had  access  to  ongoing  program  implementation  support  throughout  the  school  year  from 
behavioral  consultants.  For  this  study,  the  behavioral  consultants  were  master’s-level  therapists  or  advanced 
graduate  students.  On  average,  behavioral  consultants  were  at  the  participating  schools  for  20  hours  each 
week,  including  scheduled  activities  such  as  administrative  meetings,  in-service  presentations,  and  parent 
conferences.  Consultation  visits  typically  included  classroom  observations,  an  opportunity  for  teachers  to  ask 
questions  about  the  curriculum,  and  implementation  feedback  from  the  consultant.  In  addition  to  the 
consultation  visits,  behavioral  consultants  and  ABC  staff  provided  approximately  15  hours  per  school  of 
formal  in-service  training  and  group  presentations  throughout  the  year. 

Data  Collection 

MPR’s  subcontractor,  Decision  Information  Resources,  Inc.  (DIR),  collected  the  child,  teacher,  and  school 
data  at  the  Buffalo  site.  Table  2.3  presents  the  data  collection  dates.  Data  were  collected  in  the  fall  and  spring 
of  the  first  2 years  and  the  spring  of  Year  3.  The  fall  2004  (third-grade)  data  collection  window  began  on 
October  18,  2004,  and  ended  on  November  12,  2004.  The  average  time  frame  from  the  beginning  of  program 
implementation  to  the  beginning  of  fall  data  collection  was  6 weeks.  As  a result,  initial  data  collection  took 
place  after  ABC  implementation  began.  Therefore,  these  data  provide  a measure  of  the  students,  teachers, 
and  schools  near  the  beginning  of  the  school  year,  when  ABC  had  been  operating  for  a relatively  short  period 
of  time. 

The  spring  data  collection  window  was  from  April  11,  2005,  to  May  6,  2005.  ABC  had  been  implemented  for 
3 1 weeks  at  the  time  of  the  spring  data  collection  and  for  2 1 weeks  from  the  end  of  the  fall  data  collection. 
Year  2 followed  a similar  pattern,  with  implementation  occurring  at  the  start  of  the  school  year,  fall  data 
collection  occurring  5 weeks  later,  and  spring  data  collection  occurring  21  weeks  after  fall  data  collection  (and 
29  weeks  after  the  start  of  implementation).  In  spring  2007,  data  collection  occurred  32  weeks  after  the  start 
of  implementation.  Data  collection  took  from  3 to  5 weeks  at  each  collection  point. 


169 


Chapter  2.  Academic  and  Behavioral  Competencies  Program 


Table  2.3.  Data  collection  dates — ABC 


Year  1 Year  1 Year  2 Year  2 Year  3 

Data  collection  schedule  (Fall  3rd  grade)  (Spring  3rd  grade)  (Fall  4th  grade)  (Spring  4th  grade) (Spring  5th  grade) 


School  sample  size 

12 

12 

12 

12 

12 

School  year  dates 
First  day  of  school 

9/7/04 

t 

8/15/05 

9/7/052 

t 

8/1 5/05 
9/6/062 

Start  of  implementation 

9/8/04 

t 

First  day 

t 

First  day 

Last  day  of  school 

t 

6/24/05 

t 

6/23/06 

6/22/07 

Data  collection 

Start 

10/18/041 

4/11/05 

10/10/052 

3/27/06 

4/1 8/07 

End 

11/12/04 

5/6/05 

1 0/27/05 

5/1/06 

5/8/07 

Calendar  weeks  from 
program  implementation 
to  start  of  fall  2004 
data  collection 

6 

t 

t 

t 

t 

Calendar  weeks  from 
start  of  school  to  start  of 
fall  2004  data  collection 

6 

t 

t 

52 

t 

Calendar  weeks  from 
end  of  fall  data  collection 
to  start  spring  data 
collection 

t 

21 

t 

21 

t 

Calendar  weeks  from 
program  implementation 
to  start  of  spring  data 
collection 

t 

31 

t 

29 

32 

t Not  applicable. 

1 Data  collection  occurred  at  two  schools  on  9/21/04.  The  calculations  that  include  the  start  of  fall  2004  data  collection  use  the 
10/18/04  date  as  the  start  date  of  the  data  collection. 

2 The  two  charter  schools  started  earlier  than  the  public  schools  and  had  earlier  data  collection. 

NOTE:  The  dates  provided  in  the  table  reflect  the  more  common  dates  and  are  used  to  calculate  the  week’s  data. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Consent  Rates,  Completion  Rates,  and  Percentage  of  Sample  With  Data 

The  actual  number  of  child,  primary  caregiver,  and  teacher  reports  available  for  analysis  was  smaller  than  the 
number  in  the  sample  because  consent  and  completion  rates  were  less  than  100  percent.  Primary  caregivers 
had  to  provide  consent  before  children  could  complete  the  Child  Report,  before  their  child’s  teacher  could 
complete  the  Teacher  Report  on  Student,  and  before  they  themselves  completed  the  Primary  Caregiver 
Report.  Teachers  also  had  to  provide  consent  before  completing  the  Teacher  Report  on  Classroom  and 
School.  In  addition,  of  those  with  consent,  not  all  completed  their  respective  reports.  Table  2.4  shows  the 
consent  rates,  completion  rates,  and  percentages  of  sample  with  data  for  each  of  the  four  reports  over  the  3 
years.  For  the  Child  Report  and  the  two  teacher  reports,  completion  rates  ranged  from  91  percent  to  100 
percent.  For  the  Primary  Caregiver  Report,  the  completion  rates  dropped  over  time  from  90  percent  to  64 
percent. 

The  percentages  of  the  sample  with  Child  Report  data  ranged  from  57  percent  to  69  percent  over  the  3 years, 
with  significantly  higher  percentages  of  treatment  students  having  data  in  Years  1 and  3.  The  percentages  of 
students  with  information  from  the  Teacher  Report  on  Student  ranged  from  56  percent  to  74  percent,  with 
significantly  higher  percentages  of  treatment  students  having  data  in  Years  1 and  3.  The  percentages  of 
students  with  data  from  the  Primary  Caregiver  Report  ranged  from  42  percent  to  60  percent  and  declined 
over  time.  The  percentages  of  teachers  with  data  from  the  Teacher  Report  on  Classroom  and  School  ranged 
from  78  percent  to  92  percent. 
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Table  2.4.  Consent  rates,  completion  rates,  and  percentage  of  sample  with  data  from  each  report — ABC 


Year  1 Year  1 Year  2 Year  3 

(Fall  3rd  grade)  (Spring  3rd  grade)  (Spring  4th  grade)  (Spring  5th  grade) 

Treat-  Treat-  Treat-  Treat- 


Report 

Total 

ment 

Control 

Total 

ment 

Control 

Total 

ment 

Control 

Total 

ment 

Control 

Student  sample  size 

879 

380 

499 

875 

373 

502 

877 

367 

510 

871 

353 

518 

Child  Report  (percent) 

Primary  caregiver  consent  rate 

63.7 

68.7* 

59.7 

64.0 

67.3 

61.6 

64.7 

65.4 

64.1 

67.7 

73.7** 

63.7 

Student  completion  rate 

93.2 

91.2 

95.0 

97.5 

98.0 

97.1 

96.3 

95.8 

96.6 

93.6 

93.8 

93.3 

Students  with  data1 

59.4 

62.7* 

56.7 

62.5 

66.0 

60.0 

62.3 

62.7 

62.0 

63.4 

69.1** 

59.5 

Primary  Caregiver  Report  (percent) 

Primary  caregiver  consent  rate 

63.3 

68.2* 

59.5 

63.9 

67.3 

61.4 

64.7 

65.4 

64.1 

67.7 

73.7** 

63.7 

Primary  caregiver  completion  rate 

89.4 

88.0 

90.6 

75.3 

78.1 

73.1 

70.9 

70.0 

71.6 

65.4 

64.2 

66.4 

Primary  caregivers  with  data1 

56.6 

60.0 

53.9 

48.1 

52.5 

44.8 

45.8 

45.8 

45.9 

44.3 

47.3 

42.3 

Teacher  Report  on  Student  (percent) 

Primary  caregiver  consent  rate 2 

63.7 

68.7* 

59.7 

64.0 

67.3 

61.6 

64.7 

65.4 

64.1 

67.7 

73.7** 

63.7 

Teacher  completion  rate 

93.9 

95.0 

93.0 

99.3 

99.2 

99.4 

100.0 

100.0 

100.0 

99.2 

100.0* 

98.5 

Students  with  data1 

59.7 

65.3** 

55.5 

63.7 

66.8 

61.4 

64.7 

65.4 

64.1 

67.2 

73.7* 

62.7 

Teacher  Report  on  Classroom  and  School 
(3rd-  to  5th-grade  teachers)  (percent) 

Teacher  consent  rate 

89.9 

96.7* 

84.6 

92.1 

91.8 

92.3 

91.9 

93.2 

90.9 

85.8 

90.0 

82.7 

Teacher  completion  rate 

95.2 

94.9 

95.5 

91.4 

91.1 

91.7 

93.6 

90.9 

95.7 

95.0 

96.3 

94.0 

Teachers  with  data1 

85.6 

91.8 

80.8 

84.2 

83.6 

84.6 

86.0 

84.7 

87.0 

81.6 

86.7 

77.8 

* Treatment  group  significantly  different  from  control  group  at  the  .05  level. 

**  Treatment  group  significantly  different  from  control  group  at  the  .01  level. 

1 Calculated  as  consent  rate  x completion  rate. 

2The  primary  caregiver  consent  rates  for  the  Child  Report  and  the  Teacher  Report  on  Student  are  identical,  as  the  primary  caregiver  gave  consent  to  both  together. 
SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 


Chapter  2.  Academic  and  Behavioral  Competencies  Program 


Chapter  2.  Academic  and  Behavioral  Competencies  Program 


Responses  from  students  in  the  original  cohort  (stayers)  and  new  entrants  in  the  ABC  sample  were  examined 
to  investigate  possible  differences  between  the  two  groups  in  consent  rates,  completion  rates,  and  the 
percentages  of  sample  with  data  that  might  affect  outcome  data  (table  2.5).  There  were  four  statistically 
significant  differences  between  the  completion  rates  of  these  two  groups  across  the  3 years:  in  Year  1,  stayers 
completed  more  Child  Reports  than  did  new  entrants  (98%  versus  85%);  in  Years  2 and  3,  new  entrants 
completed  more  Primary  Caregiver  Reports  than  did  stayers  (84%  versus  68%  and  75%  versus  62%, 
respectively);  and  in  Year  3,  teachers  of  stayers  completed  more  Teacher  Reports  on  Students  than  did 
teachers  of  new  entrants  (100%  versus  97%). 
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Table  2.5.  Consent  rates,  completion  rates,  and  percentage  of  sample  with  data:  Stayers  versus  new  entrants — ABC 


Year  1 Year  2 Year  3 

(Spring  3rd  grade) (Spring  4th  grade)  (Spring  5th  grade) 


Report 

Total 

Stayers 

New 

entrants 

Total 

Stayers 

New 

entrants 

Total 

Stayers 

New 

entrants 

Student  sample  size 

875 

836 

39 

877 

719 

158 

871 

590 

281 

Child  Report  (percent) 

Primary  caregiver  consent  rate 

64.0 

63.8 

69.2 

64.7 

65.2 

62.0 

67.7 

68.8 

65.5 

Student  completion  rate 

97.5 

98.1*** 

85.2 

96.3 

96.4 

95.9 

93.6 

93.3 

94.0 

Students  with  data1 

62.4 

62.6 

59.0 

62.3 

62.9 

59.5 

63.4 

64.2 

61.6 

Primary  Caregiver  Report  (percent) 

Primary  caregiver  consent  rate 

63.9 

63.8 

66.7 

64.7 

65.2 

62.0 

67.7 

68.8 

65.5 

Primary  caregiver  completion  rate 

75.3 

76.0 

61.5 

70.9 

68.2** 

83.7 

65.8 

61.8** 

74.5 

Primary  caregivers  with  data1 

48.1 

48.4 

41.0 

45.8 

44.5 

51.9 

44.5 

42.5 

48.8 

Teacher  Report  on  Student  (percent) 

Primary  caregiver  consent  rate2 

64.0 

63.8 

69.2 

64.7 

65.2 

62.0 

67.7 

68.8 

65.5 

Teacher  completion  rate 

99.3 

99.4 

96.3 

100.0 

100.0 

100.0 

99.2 

100.0*** 

97.3 

Students  with  data1 

63.5 

63.4 

66.7 

64.7 

65.2 

62.0 

67.2 

68.8 

63.7 

* Stayers  significantly  different  from  new  entrants  at  the  .05  level. 

**  Stayers  significantly  different  from  new  entrants  at  the  .01  level. 

***  Stayers  significantly  different  from  new  entrants  at  the  .001  level. 

1 Calculated  as  consent  rate  x completion  rate. 

2The  primary  caregiver  consent  rates  for  the  Child  Report  and  the  Teacher  Report  on  Student  are  identical,  as  the  primary  caregiver  gave  consent  to  both  together. 
SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Fidelity  of  Implementation 

Each  year,  the  six  ABC  treatment  schools  were  independently  rated  for  quantity  and  quality  of  program 
implementation  by  two  raters  from  the  research  team.  The  global  measure  of  fidelity  for  the  multisite  study 
was  used;  inter-rater  reliability  was  measured  with  Cronbach’s  alpha  (1.00,  0.98,  and  0.99)  in  all  3 years.  The 
ratings  were  combined  into  a single  consensus  rating  and  used  to  identify  schools  with  high  implementation 
fidelity.  In  Year  1,  four  treatment  schools  were  identified  as  having  high  fidelity,  and  in  Years  2 and  3,  three 
schools  were  identified  as  having  high  fidelity.  Cohen’s  kappa  was  used  as  the  measure  of  agreement  when 
identifying  schools  as  high  fidelity,  and  it  equaled  1.00  in  Years  1 and  2,  and  0.67  in  Year  3. 

Initial  Characteristics 

The  initial  characteristics  of  the  students,  teachers,  and  schools  participating  in  the  evaluation  of  the  ABC 
program  were  collected  from  students  enrolled  in  the  third  grade  at  the  study  schools  in  fall  2004,  as  well  as 
from  their  primary  caregivers  and  third-grade  teachers.  In  addition,  third-,  fourth-,  and  fifth-grade  teachers 
and  principals  in  the  study  schools  provided  information  about  activities  related  to  social  and  character 
development  in  these  schools.  Documenting  the  characteristics  of  students,  teachers,  and  schools,  and  their 
status  on  key  outcomes  at  a point  before  the  intervention  had  been  operating  for  an  extended  period,  helps  to 
determine  whether  the  random  assignment  of  schools  to  treatment  and  control  status  produces  groups  with 
similar  distributions  of  observed  characteristics.  As  noted  in  the  following  discussion,  6 significant  differences 
(out  of  62  comparisons,  with  3 expected  to  be  significant  by  chance)  were  found  between  the  treatment  and 
control  students,  teachers,  and  schools  in  the  level  of  SACD  activity  in  the  classroom  and  school. 

Characteristics  of  Children,  Their  Families,  and  Communities 

There  were  no  significant  differences  in  the  observed  student,  caregiver,  and  community  characteristics 
between  the  treatment  and  control  groups  (table  2.6).  For  students,  the  mean  age  was  8 years.  The  sample 
contained  almost  equal  percentages  of  girls  and  boys.  The  sample  was  ethnically  diverse:  White  non-Hispanic 
students  made  up  33  percent  of  the  sample,  Black  non-Hispanic  students  made  up  41  percent,  and  Hispanic 
students  made  up  1 7 percent. 

The  sample  was  also  diverse  in  levels  of  family  income,  education  levels  of  primary  caregivers  of  the  children 
in  the  sample,  and  family  situation.  Fifty  percent  of  the  children  lived  in  a household  with  an  income  at  135 
percent  of  the  federal  poverty  level  or  lower,  which  is  the  income  threshold  for  eligibility  for  free  school 
meals.  Thirteen  percent  of  primary  caregivers  did  not  complete  high  school.  Half  of  the  children  lived  with 
both  their  father  and  their  mother.  There  were  no  significant  differences  between  the  treatment  and  control 
groups  in  these  characteristics. 

The  mean  values  of  the  outcomes  for  children’s  behavior  and  attitudes  as  reported  by  the  primary  caregivers, 
children,  and  teachers  at  initial  data  collection  in  fall  2004  are  shown  in  table  2.7.  There  were  no  significant 
differences  between  the  treatment  and  control  groups  in  these  scores. 
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Table  2.6.  Initial  characteristics  of  children,  their  families,  and  communities — ABC 


Characteristic 

Total 

Treatment 

Control 

Student  sample  size 

497 

228 

269 

Student  demographics 

Gender  (percent) 

Male 

50.1 

46.4 

53.9 

Female 

49.9 

53.6 

46.1 

Race/ethnicity  (percent) 

White  (non-Hispanic) 

32.7 

26.2 

39.2 

Black  (non-Hispanic) 

41.2 

48.6 

33.8 

Hispanic 

17.3 

16.9 

17.6 

Other 

8.8 

8.3 

9.4 

Age  (in  years)  (mean) 

8.0 

8.0 

8.0 

Primary  caregiver  and  family  characteristics 

Primary  caregiver’s  age  (in  years)  (mean) 

35.3 

35.9 

34.7 

Primary  caregiver’s  race/ethnicity  (percent) 

White  (non-Hispanic) 

38.3 

31.3 

45.3 

Black  (non-Hispanic) 

41.1 

46.7 

35.6 

Hispanic 

14.9 

15.4 

14.3 

Other 

5.7 

6.6 

4.8 

Primary  caregiver’s  education  (percent) 

Did  not  complete  high  school 

12.5 

11.0 

14.0 

Completed  high  school  or  equivalent 

26.6 

24.9 

28.2 

Some  college 

49.4 

52.6 

46.2 

Bachelor’s  or  higher  degree 

11.5 

11.4 

11.7 

Primary  caregiver’s  employment  (percent) 

Full-time 

45.1 

43.0 

47.1 

Other 

54.9 

57.0 

52.9 

Primary  caregiver’s  marital  status  (percent) 

Married 

47.3 

43.5 

51.2 

Other 

52.7 

56.5 

48.8 

Students  who  live  in  one  household  (percent) 

93.8 

94.0 

93.6 

Number  of  individuals  in  household  (mean) 

4.6 

4.5 

4.7 

Primary  caregiver’s  relationship  to  child  (percent) 

Mother  (stepmother) 

86.0 

86.7 

85.3 

Father  (stepfather) 

7.7 

6.0 

9.5 

Other  relative/nonrelative 

6.3 

7.4 

5.3 

See  notes  at  end  of  table. 
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Table  2.6.  Initial  characteristics  of  children,  their  families,  and  communities — ABC — Continued 


Characteristic 

Total 

Treatment 

Control 

Student  lives  with  (percent) 

Mother  (stepmother)  and  father  (stepfather) 

50.4 

46.6 

54.2 

Mother  (stepmother)  only;  father  (stepfather)  not  present 

42.5 

44.8 

40.3 

Father  (stepfather)  only;  mother  (stepmother)  not  present 

1.8 

2.1 

1.6 

Other  relative/nonrelative,  parents  not  present 

5.2 

6.6 

3.9 

Highest  education  of  anyone  in  household  (percent) 

Did  not  complete  high  school 

8.7 

5.2 

12.1 

Completed  high  school  or  equivalent 

23.3 

22.5 

24.0 

Some  college 

52.4 

56.6 

48.2 

Bachelor’s  or  higher  degree 

15.6 

15.7 

15.6 

Total  household  income  (percent) 

Less  than  $20,000 

40.7 

39.8 

41.6 

$20,000  to  $39,999 

34.0 

35.5 

32.4 

$40,000  to  $59,999 

16.2 

17.9 

14.5 

$60,000  or  more 

9.1 

6.7 

11.5 

Income-to-poverty-threshold  ratio — Below  135  percent  (percent) 

49.7 

48.0 

51.4 

Income-to-poverty-threshold  ratio — 135  to  185  percent  (percent) 

25.2 

26.9 

23.4 

Income-to-poverty-threshold  ratio — Above  185  percent  (percent) 

25.2 

25.1 

25.3 

Alabama  Parenting  Questionnaire — Poor  Monitoring  and 

Supervision  Subscale  (mean) 

1.1 

1.2 

1.1 

Alabama  Parenting  Questionnaire — Positive  Parenting 

Subscale  (mean) 

3.5 

3.5 

3.5 

Confusion,  Hubbub,  and  Order  Scale  (mean) 

2.2 

2.2 

2.2 

Community  characteristics  (mean) 

Community  Risks  Scale 

1.7 

1.7 

1.7 

Community  Resources  Scale 

2.8 

2.8 

2.8 

Child-Centered  Social  Control  Scale 

3.0 

3.0 

3.0 

NOTE:  No  statistically  significant  differences  were  found  between  values  for  treatment  and  control  groups.  Weights,  which  assign 
equal  weight  to  each  school  within  the  program,  were  used  in  producing  the  treatment,  control,  and  overall  means.  Statistical  tests 
were  conducted  using  regressions  that  included  program  indicators  to  account  for  the  sample  design  and  adjusted  for  clustering  at 
the  school  level. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Table  2.7.  Mean  scores  and  standard  deviations  for  initial  outcome  measures  of  sample — ABC 


Outcome  measure-Report 

Range 

Total 

Mean 

SD 

Treatment 
Mean  SD 

Control 
Mean  SD 

Social  and  Emotional  Competence  Domain 

Self-Efficacy  for  Peer  Interaction-CR 

1-4 

3.0 

0.7 

3.0 

0.7 

3.0 

0.7 

Normative  Beliefs  About  Aggression-CR 

1-4 

1.3 

0.5 

1.2 

0.5 

1.3 

0.5 

Empathy-CR 

1-3 

2.3 

0.4 

2.3 

0.4 

2.4 

0.4 

Behavior  Domain 

Altruistic  Behavior-CR 

0-3 

1.5 

0.9 

1.5 

0.9 

1.5 

0.9 

Altruistic  Behavior-TRS 

1-4 

1.3 

0.4 

1.4 

0.4 

1.3 

0.4 

Altruistic  Behavior-PCR 

1-4 

2.3 

0.7 

2.3 

0.7 

2.3 

0.7 

Positive  Social  Behavior-TRS 

1-4 

2.9 

0.7 

2.9 

0.7 

2.9 

0.7 

Positive  Social  Behavior-PCR 

1-4 

2.9 

0.5 

2.9 

0.5 

3.0 

0.5 

Problem  Behavior-CR 

0-3 

0.3 

0.5 

0.3 

0.5 

0.3 

0.5 

Problem  Behavior-TRS 

1-4 

1.4 

0.5 

1.4 

0.4 

1.4 

0.5 

Problem  Behavior-PCR 

1-4 

1.6 

0.3 

1.6 

0.3 

1.6 

0.4 

ADHD-Related  Behavior-TRS 

1-4 

1.8 

0.6 

1.8 

0.6 

1.8 

0.6 

Academics  Domain 

Academic  Competence  and  Motivation-TRS 

1-5 

2.8 

0.8 

2.7 

0.9 

2.8 

0.8 

Engagement  with  Learning-CR 

1-4 

3.7 

0.6 

3.6 

0.7 

3.7 

0.6 

Perceptions  of  School  Climate  Domain 

Positive  School  Orientation-CR 

1-4 

3.2 

0.7 

3.2 

0.7 

3.3 

0.7 

Negative  School  Orientation-CR 

1-4 

1.9 

0.7 

2.0 

0.7 

1.9 

0.7 

Student  Afraid  at  School-CR 

1-4 

2.3 

1.0 

2.4 

1.0 

2.3 

1.0 

Victimization  at  School-CR 

0-3 

0.8 

0.8 

0.7 

0.8 

0.8 

0.8 

Student  sample  size — PCR 

497 

228 

269 

Student  sample  size — CR 

521 

238 

283 

Student  sample  size — TRS 

525 

248 

277 

NOTE:  Abbreviations  are 


CR:  Child  Report 

PCR:  Primary  Caregiver  Report 

TRS:  Teacher  Report  on  Student 

ADHD:  Attention  deficit  hyperactivity  disorder 

SD:  Standard  deviation 

No  statistically  significant  differences  were  found  between  values  for  treatment  and  control  groups.  Weights,  which  assign  equal 
weight  to  each  school  within  the  program,  were  used  in  producing  the  treatment,  control,  and  overall  means.  Statistical  tests  were 
conducted  using  regressions  that  included  program  indicators  to  account  for  the  sample  design  and  adjusted  for  clustering  at  the 
school  level.  Sample  size  may  differ  for  some  outcomes  due  to  nonresponse. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Characteristics  of  Teachers  and  Schools 

The  third-,  fourth-,  and  fifth-grade  teachers  at  the  study  schools  were  predominandy  White  non-Hispanic 
(80%)  and  female  (90%),  had  an  average  of  12.3  years  of  total  teaching  experience,  and  about  three-fourths 
(77%)  held  an  advanced  or  specialist  degree  (table  2.8).  There  were  no  statistically  significant  differences 
between  the  treatment  and  control  groups  of  teachers. 

Data  regarding  school  characteristics  were  drawn  from  the  Common  Core  of  Data  in  order  to  compare 
treatment  and  control  schools.  There  were  no  significant  differences  between  the  two  groups  of  schools  in 
terms  of  student  composition  (race/ ethnicity  and  school  lunch  eligibility),  number  of  students  enrolled, 
number  of  full-time  teachers,  Title  I status,  or  number  of  years  the  principal  had  been  at  the  school  (see  table 
2.9).  In  addition,  there  were  no  significant  differences  between  treatment  and  control  schools  in  terms  of 
location  (urban,  suburban,  or  rural)  or  lowest  and  highest  grade  offered  (these  data  are  not  shown  in  a table). 
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Table  2.8.  Initial  characteristics  of  teachers  in  sample — ABC 


Characteristic 

Total 

Treatment 

Control 

Teacher  sample  size 

119 

56 

63 

Gender  (percent) 

Male 

9.7 

9.4 

10.1 

Female 

90.3 

90.6 

89.9 

Race/ethnicity  (percent) 

White  (non-Hispanic) 

80.1 

73.2 

86.9 

Other 

19.9 

26.8 

13.1 

Number  of  years  teaching  experience  (mean) 

12.3 

12.7 

11.9 

Number  of  years  teaching  experience  in  this  school  (mean) 

4.9 

4.5 

5.3 

Type  of  teaching  certificate  (percent) 

Regular  state  certificate  or  advanced  professional  certificate 

65.7 

63.4 

68.0 

Other 

34.3 

36.6 

32.0 

Education  (percent) 

Bachelor’s  degree 

23.0 

24.0 

22.0 

Advanced  degree/other 

77.0 

76.0 

78.0 

NOTE:  No  statistically  significant  differences  were  found  between  values  for  treatment  and  control  groups.  Weights,  which  assign 
equal  weight  to  each  school  within  the  program,  were  used  in  producing  the  treatment,  control,  and  overall  means.  Statistical  tests 
were  conducted  using  regressions  that  included  program  indicators  to  account  for  the  sample  design  and  adjusted  for  clustering  at 
the  school  level.  Sample  size  may  differ  for  some  outcomes  due  to  nonresponse. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Table  2.9.  Initial  characteristics  of  schools  in  sample — ABC 


Characteristic 

Total 

Treatment 

Control 

School  sample  size 

12 

6 

6 

Student  race/ethnicity  (percent) 

White  (non-Hispanic) 

34.3 

29.1 

39.6 

Black  (non-Hispanic) 

48.8 

54.9 

42.7 

Hispanic 

14.8 

14.6 

15.0 

Other 

2.0 

1.4 

2.7 

Students  eligible  for  free  or  reduced-price  lunch  (percent) 

66.7 

68.5 

65.3 

Number  of  students  enrolled  (mean) 

679.2 

584.3 

774.0 

Number  of  full-time  teachers  (mean) 

46.5 

41.6 

51.5 

Title  1 status  (percent) 

Title  1 eligible  school 

83.3 

83.3 

83.3 

Schoolwide  Title  1 

90.0 

80.0 

100.0 

Number  of  years  principal  has  been  at  this  school  (mean) 

4.3 

4.5 

4.2 

NOTE:  No  statistically  significant  differences  were  found  between  values  for  treatment  and  control  groups.  Weights,  which  assign 
equal  weight  to  each  school  within  the  program,  were  used  in  producing  the  treatment,  control,  and  overall  means.  Statistical  tests 
were  conducted  using  regressions  that  included  program  indicators  to  account  for  the  sample  design  and  adjusted  for  clustering  at 
the  school  level.  Sample  size  may  differ  for  some  outcomes  due  to  nonresponse. 

SOURCE:  NCES  Common  Core  of  Data  (2003-04),  the  Social  and  Character  Development  (SACD)  Research  Program. 

In  the  Teacher  Report  on  Classroom  and  School,  teachers  reported  on  nine  dimensions  of  school 
environment  (these  data  are  not  shown  in  a table):  feelings  of  safety,  adequacy  of  resources,  student  support, 
freedom  to  teach  as  desired,  affiliation  with  and  ties  to  colleagues,  innovation  regarding  new  approaches  to 
teaching,  professional  interest,  participatory  decisionmaking,  and  work  pressure.  There  were  no  statistically 
significant  differences  between  treatment  and  control  schools  in  these  reports. 

The  Level  of  SACD  in  the  Schools  Near  the  Beginning  of  the  Study 

During  the  initial  data  collection  period,  principals  and  teachers  reported  on  the  SACD  activities  used  in  the 
schools  and  classrooms,  the  availability  of  SACD  materials,  and  the  professional  development  provided  on 
SACD  (see  tables  2.10,  2.11,  and  2.12,  respectively).  It  is  important  to  note  that  data  collection  at  the 
beginning  of  the  school  year  took  place  6 weeks  after  the  start  of  ABC  implementation;  therefore  the  SACD- 
related  activities  and  practices  in  the  treatment  and  control  schools  are  a measure  of  what  was  happening  in 
the  study  schools  near  the  beginning  of  the  study.  The  principals  at  the  12  schools  reported  on  activities  to 
promote  six  social  and  character  development  goals  (see  table  2.10):  violence  prevention  and  peace 
promotion  (83%),  social  and  emotional  development  (92%),  character  education  (100%),  tolerance  and 
diversity  (75%),  risk  prevention  and  health  promotion  (83%),  and  civic  responsibility  and  community  service 
(83%).  In  addition,  all  12  principals  reported  activities  directed  toward  behavior  management.  There  were  no 
statistically  significant  differences  between  the  treatment  and  control  schools  in  the  percentages  of  principal 
reports,  although  this  may  be  due  to  the  relatively  small  principal  sample  size.  The  percentages  of  teachers 
who  reported  the  use  of  these  activities  in  their  classrooms  ranged  from  63  percent  to  93  percent;  there  were 
no  significant  differences  between  treatment  and  control  teachers.  With  respect  to  the  use  of  schoolwide 
activities,  59  percent  to  81  percent  of  teachers  reported  that  their  schools  used  such  activities,  and  there  were 
no  significant  differences  between  treatment  and  control  teachers  in  reported  use. 
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Table  2.10.  Principal  and  teacher  initial  reports  on  use  of  SACD  programs  or  activities  in 
sample — ABC 


SACD  program  or  activity 

Total 

Treatment 

Control 

Principal  sample  size 

12 

6 

6 

Teacher  sample  size 

119 

56 

63 

Principals  reporting  that  school  had  programs  or  activities 
to  promote  the  following  SACD  goals  (percent) 

Violence  prevention  and  peace  promotion 

83.3 

66.7 

100.0 

Social  and  emotional  development 

91.7 

83.3 

100.0 

Character  education 

100.0 

100.0 

100.0 

Tolerance  and  diversity 

75.0 

66.7 

83.3 

Risk  prevention  and  health  promotion 

83.3 

83.3 

83.3 

Civic  responsibility  and  community  service 

83.3 

83.3 

83.3 

Behavior  management 

100.0 

100.0 

100.0 

None  of  the  above 

0.0 

0.0 

0.0 

Teachers  reporting  on  using  programs  or  activities  in  their 
class  to  promote  the  following  SACD  goals  (percent) 

Violence  prevention  and  peace  promotion 

74.7 

82.1 

67.2 

Social  and  emotional  development 

81.9 

87.1 

76.8 

Character  education 

88.5 

93.1 

83.8 

Tolerance  and  diversity 

71.6 

70.2 

73.0 

Risk  prevention  and  health  promotion 

64.1 

66.7 

61.6 

Civic  responsibility  and  community  service 

63.0 

71.1 

54.9 

Behavior  management 

93.4 

98.6 

88.2 

None  of  the  above 

0.0 

0.0 

0.0 

Teachers  reporting  schoolwide  use  of  the  following 
activities  to  promote  SACD  (percent) 

Morning  announcements  or  videos 

80.9 

81.1 

80.8 

School  assemblies 

76.2 

70.4 

82.1 

School  newspapers  or  bulletins 

64.4 

67.6 

61.3 

Special  school  days 

59.2 

66.6 

51.8 

Special  events 

78.4 

80.3 

76.6 

Other  activities 

17.5 

19.4 

15.6 

NOTE:  No  statistically  significant  differences  were  found  between  values  for  treatment  and  control  groups.  Weights,  which  assign 
equal  weight  to  each  school  within  each  of  the  programs  and  to  each  program  across  programs,  were  used  in  producing  the 
treatment,  control,  and  overall  means.  Statistical  tests  were  conducted  using  regressions  that  included  program  indicators  to 
account  for  the  sample  design  and  adjusted  for  clustering  at  the  school  level.  Sample  size  may  differ  for  some  outcomes  due  to 
nonresponse. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Teachers  reported  using  a broad  range  of  teaching  materials  (table  2.11).  The  materials  most  used  in  support 
of  SACD  activities  were  teacher  guides  (used  by  59%  of  teachers),  giveaways  (used  by  55%),  children’s 
literature  (used  by  47%),  and  student  materials  (used  by  46%).  There  were  no  significant  differences  between 
treatment  and  control  teachers  in  the  use  of  materials. 

Teachers  reported  using  a wide  variety  of  teaching  strategies  (table  2.11).  All  teachers  in  the  sample  reported 
using  any  of  the  20  strategies,  and,  on  average,  they  used  12  of  the  strategies.  There  were  no  significant 
differences  between  treatment  and  control  teachers  in  the  average  number  of  strategies  used.  In  regard  to  the 
specific  strategies,  there  were  5 significant  differences  out  of  29  comparisons  made  (with  1 expected  to  be 
significant  by  chance).  A significantly  greater  number  of  treatment  teachers  reported  incorporating  SACD 
into  the  academic  curriculum  (87%  versus  62%),  sending  home  good  behavior  notes  (93%  versus  64%), 
presenting  role  models  (76%  versus  59%),  using  an  honor  roll  for  positive  behavior  (76%  versus  45%),  and 
using  time-outs  for  negative  behavior  (94%  versus  73%).  These  differences  between  treatment  and  control 
teachers  at  the  start  of  the  study  may  reflect  the  fact  that  program  implementation  at  the  treatment  schools 
began  before  data  collection  at  the  start  of  the  school  year. 
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Table  2.11.  Teacher  initial  reports  on  use  of  SACD  materials  and  classroom  strategies  in 
sample — ABC 


SACD  material  and  classroom  strategy 

Total 

Treatment 

Control 

Teacher  sample  size 

119 

56 

63 

Teachers  using  the  following  materials  in  conjunction  with 
social  and  character  development  activities  (percent) 

Teacher  guides  (manuals,  curricula) 

59.4 

48.1 

70.7 

Student  materials  (workbooks,  worksheets) 

46.2 

46.1 

46.2 

Instructional  aids  (games,  software,  videos) 

26.8 

29.3 

24.3 

Giveaways  (bookmarks,  stickers) 

55.3 

59.5 

51.2 

Children’s  literature 

47.3 

52.3 

42.3 

Other  types  of  materials 

11.4 

11.2 

11.7 

Do  not  use  any  of  the  materials  listed  above 

13.3 

14.3 

12.4 

Teachers  using  any  of  the  strategies  listed  below  to  promote 
social  and  character  development  in  the  classroom  (percent) 

100.0 

100.0 

100.0 

Number  of  strategies  (listed  below)  used  by  teachers  to  promote 
social  and  character  development  in  the  classroom  (mean) 

11.7 

12.7 

10.8 

Teachers  using  each  of  the  following  strategies  to  promote 
social  and  character  development  (percent) 

Role-playing 

67.1 

63.1 

71.1 

Cooperative  learning 

99.4 

100.0 

98.7 

Peer  group  discussions 

86.3 

91.4 

81.1 

Direct  instruction  of  social  and  character  development 

82.9 

88.8 

77.0 

Skill  training 

47.6 

46.2 

49.0 

Incorporating  social  and  character  development  into 
academic  curriculum 

74.6 

87.2** 

62.0 

Parent  training 

6.0 

5.1 

6.8 

Parent/community  involvement  in  program  development 
or  delivery 

19.4 

19.6 

19.2 

Mentoring 

30.9 

36.1 

25.6 

Good  behavior  notes  sent  home  daily  or  weekly 

78.2 

92.6* 

63.9 

Presenting  role  models 

67.5 

75.6* 

59.3 

Targeted  story  reading  or  writing  on  SACD  themes 

72.6 

77.3 

67.9 

Peer  mediation 

36.0 

36.9 

35.1 

Honor  roll  for  positive  behavior 

60.0 

75.5* 

44.6 

Pledges  or  recitations  on  social  and  character 
development  themes 

41.1 

37.8 

44.4 

Guided  visualization 

45.1 

49.9 

40.3 

Student-led/student-assisted  instruction 

42.3 

41.4 

43.2 

Journaling 

70.5 

80.3 

60.7 

Time  out  for  negative  behavior 

83.6 

94.0* 

73.2 

Daily  or  weekly  rewards  for  positive  behavior 

93.7 

98.1 

89.2 

* Treatment  group  significantly  different  from  control  group  at  the  .05  level. 


**  Treatment  group  significantly  different  from  control  group  at  the  .01  level. 

NOTE:  Weights,  which  assign  equal  weight  to  each  school  within  each  of  the  programs  and  to  each  program  across  programs,  were 
used  in  producing  the  treatment,  control,  and  overall  means.  Statistical  tests  were  conducted  using  regressions  that  included 
program  indicators  to  account  for  the  sample  design  and  adjusted  for  clustering  at  the  school  level.  Sample  size  may  differ  for  some 
outcomes  due  to  nonresponse. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Principals  and  teachers  reported  on  participation  in  and  amount  of  SACD  training  and  staff  development 
provided  over  the  previous  12  months  (table  2.12).  Principals  reported  higher  participation  rates  than  did 
teachers  (100%  versus  76%),  and  teachers  reported  more  training  hours  (11  versus  8).  There  were  no 
significant  differences  between  treatment  and  control  principals  and  teachers  in  overall  training  and  hours. 
Out  of  seven  targeted  areas  for  professional  development,  there  was  one  statistically  significant  difference  in 
the  area  of  behavior  management,  with  more  treatment  teachers  than  control  teachers  reporting  attending 
training  in  this  area  (71%  versus  41%). 


Table  2.12.  Principal  and  teacher  initial  reports  on  SACD  professional  development  in  sample — 
ABC 


SACD  professional  development 

Total 

Treatment 

Control 

Principal  sample  size 

12 

6 

6 

Teacher  sample  size 

119 

56 

63 

Principals  reporting  that  staff  participated  in  social  and  character 
development  training  within  the  past  year  (percent) 

100.0 

100.0 

100.0 

Teachers  reporting  participation  in  social  and  character 
development  training  within  the  past  12  months  (percent) 

76.2 

81.9 

70.5 

Number  of  hours  of  social  and  character  development  training 
principals  report  were  provided  to  each  staff  person  last  year  (mean) 

7.8 

8.0 

7.5 

Number  of  hours  of  social  and  character  development  training 
teachers  report  receiving  during  the  past  12  months  (mean) 

10.8 

11.9 

9.6 

Teachers  reporting  receiving  training  in  the  past  12  months 
in  the  following  areas  (percent) 

Violence  prevention  and  peace  promotion 

22.5 

13.7 

31.2 

Social  and  emotional  development 

25.2 

23.1 

27.2 

Character  education 

37.4 

35.5 

39.3 

Tolerance  and  diversity 

20.5 

17.6 

23.5 

Risk  prevention  and  health  promotion 

6.4 

5.8 

7.0 

Civic  responsibility  and  community  service 

3.0 

t 

t 

Behavior  management 

56.0 

71.2* 

40.7 

X Reporting  standards  not  met.  Values  suppressed  to  protect  confidentiality. 
* Treatment  group  significantly  different  from  control  group  at  the  .05  level. 


NOTE:  Weights,  which  assign  equal  weight  to  each  school  within  each  of  the  programs  and  to  each  program  across  programs,  were 
used  in  producing  the  treatment,  control,  and  overall  means.  Statistical  tests  were  conducted  using  regressions  that  included 
program  indicators  to  account  for  the  sample  design  and  adjusted  for  clustering  at  the  school  level.  Sample  size  may  differ  for  some 
outcomes  due  to  nonresponse. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 


The  data  on  the  initial  level  of  SACD  activity  illustrate  that  the  control  condition  was  a “standard  practice” 
control.  Standard  practice  at  the  control  schools  included  using  SACD  activities,  materials,  and  practices, 
along  with  professional  development  for  staff,  at  rates  and  in  types  and  amounts  similar  to  the  treatment 
schools.  For  example,  the  percentages  of  teachers  who  reported  using  programs  or  activities  to  promote 
specific  SACD  goals  ranged  from  67  percent  to  99  percent  in  the  treatment  schools  and  from  55  percent  to 
88  percent  in  the  control  schools.  However,  the  6 significant  differences  between  the  treatment  and  control 
conditions  in  the  use  of  SACD  activities  were  more  than  would  be  expected  by  chance  (3  out  of  62 
comparisons),  and  in  all  cases  they  favored  the  treatment  group.  Because  initial  data  collection  happened  after 
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program  implementation  began,  these  differences  may  reflect  that  program  implementation  and  program 
training  for  staff  started  before  initial  data  collection. 

Impacts  on  Use  of  SACD  Activities 

The  introduction  of  the  formal  ABC  program  would  be  expected  to  increase  the  use  of  SACD  activities  in 
the  treatment  schools  in  comparison  to  the  control  schools.  The  analysis  of  this  impact  is  based  on  the 
Teacher  Report  on  Classroom  and  School  (TRCS).  Every  spring,  third-,  fourth-,  and  fifth-grade  teachers  in 
the  schools  provided  information  through  the  TRCS  about  the  social  and  character  development  activities 
they  used  in  their  classrooms.  Specifically,  information  from  the  TRCS  was  used  to  determine  the  difference 
between  treatment  and  control  teachers  in  the  following  areas: 

1 . the  use  of  SACD  activities  in  their  classrooms  overall  and  by  SACD  goal; 

2.  the  use  of  materials  and  strategies  to  implement  the  SACD  activities  within  classrooms  and  within 
the  entire  school; 

3.  the  use  of  staff  development  to  support  the  teachers;  and 

4.  teacher  support  for  SACD  efforts  in  the  school  and  the  use  of  practices  conducive  to  the  social  and 
character  development  of  students. 

TRCS  consent  and  completion  rates  (table  2.4)  led  to  82  percent  to  86  percent  of  teachers  overall  having  data 
for  the  3 years,  with  greater  percentages  of  treatment  teachers  providing  data  in  most  of  the  years.  To 
estimate  intervention  impacts  for  each  of  the  outcomes,  the  statistical  significance  of  the  differences  in  means 
was  assessed.  Preliminary  analysis  using  covariates  indicated  little  or  no  gains  in  precision.  Before  testing  the 
mean  differences,  the  data  were  weighted  such  that  each  school  received  equal  weight.  Standard  errors  of  the 
impact  estimates  accounted  for  the  clustering  of  teachers  within  schools.  In  addition,  a set  of  heuristics 
(described  in  chapter  1)  was  applied  to  determine  whether  each  outcome  domain  was  statistically  significant 
after  adjustments  were  made  for  the  multiple  tests  conducted. 

Use  of  Activities 

The  percentages  of  control  teachers  who  reported  using  any  SACD  activities  in  their  classrooms  ranged  from 
85  percent  to  97  percent  over  the  3 years  (table  2.13,  panel  1).  For  the  six  individual  SACD  goals,  the 
percentages  varied  from  52  percent  to  87  percent  in  Year  1,  54  percent  to  71  percent  in  Year  2,  and  42 
percent  to  69  percent  in  Year  3.  Control  teachers’  use  of  behavior  management  activities  ranged  from  80 
percent  to  86  percent  over  this  period.  The  percentages  of  control  teachers  who  reported  using  any  SACD 
activities  in  their  classrooms  for  at  least  1 hour  per  week  ranged  from  78  percent  to  80  percent  over  the  3 
years  (table  2.13,  panel  2).  For  the  six  individual  SACD  goals,  the  range  varied  from  20  percent  to  55  percent 
in  Year  1,  14  percent  to  38  percent  in  Year  2,  and  14  percent  to  35  percent  in  Year  3.  Their  use  of  behavior 
management  activities  for  at  least  1 hour  per  week  ranged  from  62  percent  to  74  percent  over  this  period. 
These  findings  show  that  the  control  schools  were  using  these  activities  as  part  of  their  standard  practice 
related  to  social  and  character  development. 

For  teachers’  reported  use  of  any  SACD  activity  (panels  1 and  2),  48  comparisons  were  made,  with  2 expected 
to  be  significant  by  chance.  The  percentages  of  treatment  teachers  using  any  SACD  activity  were  not 
significantly  different  from  the  percentages  of  control  teachers  in  any  year  (panel  1).  A significantly  larger 
percentage  of  treatment  teachers  reported  using  civic  responsibility  activities  (impact  = 23  percentage  points) 
in  Year  1.  In  Year  2,  more  treatment  teachers  reported  using  activities  for  social  and  emotional  development 
(impact  = 24  percentage  points),  character  education  (impact  = 20  percentage  points),  tolerance  and  diversity 
(impact  = 24  percentage  points),  and  behavior  management  (impact  =16  percentage  points).  There  were  no 
significant  differences  between  treatment  and  control  teacher  reports  of  using  activities  for  at  least  1 hour  a 
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week  (panel  2).  After  the  heuristics  were  applied,  the  domain  for  engagement  in  any  SACD  activities  showed 
the  ABC  program  had  statistically  significant  impacts  in  Year  1. 

For  teachers’  reported  use  of  any  named  SACD  activity  (panels  3 and  4),  42  comparisons  were  made,  with  2 
expected  to  be  significant  by  chance.  None  of  the  impact  estimates  were  statistically  significant  over  the  3 
years.  The  overall  impact  of  the  ABC  program  on  the  domain  for  engagement  in  named  SACD  activities  was 
not  significant  in  any  year. 
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Table  2.13.  Impacts  on  teacher-reported  SACD  classroom  activities — ABC 

Panel  1 : Engagement  in  any  activities  to  promote  SACD  goals1 

Year  1 Year  2 Year  3 

(Spring  3rd  grade) (Spring  4th  grade) (Spring  5th  grade) 

T reat-  T reat-  T reat- 


SACD  activity 

ment 

Control 

Impact 

p-value 

ment 

Control 

Impact 

p-value 

ment 

Control 

Impact 

p-value 

Teacher  sample  size 

63 

73 

50 

67 

51 

64 

Violence  prevention  and  peace  promotion 
(percent) 

70.3 

77.0 

-6.8 

0.587 

74.8 

63.3 

11.5 

0.339 

63.5 

62.0 

1.5 

0.882 

Social  and  emotional  development 
(percent) 

77.7 

86.5 

-8.8 

0.474 

85.6* 

61.9 

23.8 

0.044 

69.2 

55.2 

14.0 

0.111 

Character  education  (percent) 

91.4 

83.5 

7.9 

0.383 

90.9* 

71.1 

19.8 

0.044 

69.6 

69.0 

0.6 

0.961 

Tolerance  and  diversity  (percent) 

78.7 

71.1 

7.7 

0.451 

77.9* 

54.2 

23.7 

0.014 

66. 4A 

44.8 

21.6 

0.072 

Risk  prevention  and  health  promotion 
(percent) 

74.8 

80.2 

-5.4 

0.664 

65.5 

68.4 

-2.9 

0.833 

57.3 

56.6 

0.7 

0.955 

Civic  responsibility  and  community  service 
(percent) 

74.7* 

51.8 

22.9 

0.038 

62.9 

53.9 

9.1 

0.571 

65. 6A 

41.6 

24.0 

0.094 

Any  SACD  goal  (percent) 

94.0 

97.1 

-3.2 

0.454 

98.8 

92.6 

6.2 

0.108 

93.8 

84.6 

9.2 

0.214 

Behavior  management  (percent) 

92.7 

79.6 

13.1 

0.186 

96.9* 

81.1 

15.7 

0.022 

94.1 

85.8 

8.3 

0.179 

See  notes  at  end  of  table. 
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Table  2.13.  Impacts  on  teacher-reported  SACD  classroom  activities — ABC — Continued 


Panel  2:  Engagement  in  any  activities  to  promote  SACD  goals  for  at  least  1 hour  per  week 

Year  1 Year  2 Year  3 

(Spring  3rd  grade) (Spring  4th  grade) (Spring  5th  grade) 

T reat-  T reat-  T reat- 


SACD  activity 

ment 

Control 

Impact 

p-value 

ment 

Control 

Impact 

p-value 

ment 

Control 

Impact 

p-value 

Teacher  sample  size 

63 

73 

50 

67 

51 

64 

Violence  prevention  and  peace  promotion 
(percent) 

32.9 

44.0 

-11.2 

0.341 

41.0 

37.2 

3.8 

0.797 

27.5 

34.9 

-7.4 

0.537 

Social  and  emotional  development 
(percent) 

40.4 

54.5 

-14.0 

0.314 

38.2 

34.9 

3.2 

0.827 

33.2 

29.6 

3.5 

0.748 

Character  education  (percent) 

45.5 

50.2 

-4.7 

0.747 

61. 2A 

34.1 

27.0 

0.059 

38.5 

34.6 

4.0 

0.723 

Tolerance  and  diversity  (percent) 

30.8 

38.7 

-7.9 

0.568 

23.2 

22.2 

1.0 

0.913 

20.1 

19.9 

0.3 

0.968 

Risk  prevention  and  health  promotion 
(percent) 

38.9 

46.7 

-7.8 

0.433 

22.9 

37.5 

-14.6 

0.304 

17.1 

17.2 

0.0 

0.996 

Civic  responsibility  and  community  service 
(percent) 

25.4 

20.1 

5.4 

0.522 

15.6 

14.1 

1.5 

0.886 

14.0 

14.0 

0.0 

0.997 

Any  SACD  goal  (percent) 

74.8 

80.1 

-5.3 

0.560 

77.6 

78.3 

-0.7 

0.952 

84.0 

79.6 

4.4 

0.567 

Behavior  management  (percent) 

78.8 

61.5 

17.3 

0.232 

79.9 

68.7 

11.2 

0.454 

79.4 

73.8 

5.6 

0.527 

See  notes  at  end  of  table. 
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Table  2.13.  Impacts  on  teacher-reported  SACD  classroom 


Panel  3:  Engagement  in  activities  to  promote  SACD  goals  linked  to  named  SACD 

Year  1 

(Spring  3rd  grade) 


SACD  activity 

T reat- 
ment 

Control 

Impact 

p-value 

Teacher  sample  size 

63 

73 

Violence  prevention  and  peace  promotion 

(percent) 

30.8 

28.9 

1.8 

0.856 

Social  and  emotional  development 

(percent) 

28.1 

12.5 

15.6 

0.160 

Character  education  (percent) 

33.1 

20.6 

12.5 

0.317 

Tolerance  and  diversity  (percent) 

20.0 

10.1 

9.8 

0.233 

Risk  prevention  and  health  promotion 

(percent) 

21.7 

37.2 

-15.5 

0.156 

Civic  responsibility  and  community  service 

(percent) 

t 

t 

1.6 

0.717 

Any  named  activity  (percent) 

50.8 

55.2 

-4.4 

0.742 

See  notes  at  end  of  table. 
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Year  2 

(Spring  4th  grade) 

Year  3 

(Spring  5th  grade) 

T reat- 
ment 

Control 

Impact 

p-value 

T reat- 
ment 

Control 

Impact 

p-value 

50 

67 

51 

64 

26.2 

30.2 

-4.0 

0.670 

14.1 

24.9 

-10.9 

0.280 

28.4 

17.6 

10.8 

0.264 

9.8 

16.7 

-7.0 

0.474 

25.9 

13.6 

12.3 

0.219 

11.6 

13.9 

-2.3 

0.768 

19.4 

6.8 

12.6 

0.125 

8.0 

7.9 

0.2 

0.979 

21.8 

37.1 

-15.3 

0.252 

17.1 

16.3 

0.8 

0.941 

tA 

t 

9.7 

0.059 

9.8 

8.1 

1.7 

0.773 

42.7 

55.1 

-12.4 

0.244 

31.1 

46.6 

-15.6 

0.283 
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Table  2.13.  Impacts  on  teacher-reported  SACD  classroom  activities — ABC — Continued 


Panel  4:  Engagement  in  activities  to  promote  SACD  goals  linked  to  named  SACD  programs  for  at  least  1 hour  per  week 

Year  1 Year  2 Year  3 

(Spring  3rd  grade) (Spring  4th  grade) (Spring  5th  grade) 

T reat-  T reat-  T reat- 

SACD  activity ment  Control  Impact  p-value ment  Control  Impact  p-value ment  Control  Impact  p-value 


Teacher  sample  size 

63 

73 

50 

67 

51 

64 

Violence  prevention  and  peace  promotion 
(percent) 

25.2 

18.4 

6.7 

0.381 

21.8 

24.9 

-3.2 

0.735 

8.6 

13.4 

-4.8 

0.537 

Social  and  emotional  development 
(percent) 

23.6 

7.8 

15.8 

0.141 

20.7 

11.5 

9.2 

0.327 

6.3 

10.0 

-3.7 

0.553 

Character  education  (percent) 

21.9 

13.6 

8.3 

0.420 

25. 4A 

8.9 

16.5 

0.073 

7.2 

9.3 

-2.1 

0.767 

Tolerance  and  diversity  (percent) 

8.4 

4.3 

4.1 

0.397 

1 3.7A 

3.9 

9.8 

0.073 

t 

* 

-0.8 

0.760 

Risk  prevention  and  health  promotion 
(percent) 

13. 3A 

29.1 

-15.7 

0.088 

15.8 

25.5 

-9.7 

0.464 

5.3 

7.3 

-2.0 

0.706 

Civic  responsibility  and  community  service 
(percent) 

5.3 

3.5 

1.8 

0.690 

t 

i 

2.9 

0.348 

4.1 

0.0 

4.1 

t 

Any  named  activity  (percent) 

34.4 

44.5 

-10.1 

0.410 

32.2 

41.0 

-8.7 

0.498 

14.5 

20.5 

-6.0 

0.606 

t Not  applicable. 

t Reporting  standards  not  met.  Values  suppressed  to  protect  confidentiality. 

* Treatment  group  significantly  different  from  control  group  at  the  .05  level. 

A Treatment  group  significantly  different  from  control  group  at  the  .10  to  > .05  level. 

1 In  Year  1 , the  omnibus  impact  for  all  the  outcomes  measured  together  was  positive  and  statistically  significant  on  the  basis  of  a multivariate  statistical  test. 

NOTE:  Weights,  which  assign  equal  weight  to  each  school  within  the  program,  were  used  in  producing  the  treatment,  control,  and  overall  means.  The  number  of  results  found 
significant  was  no  more  than  expected  by  chance. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 


Chapter  2.  Academic  and  Behavioral  Competencies  Program 


Chapter  2.  Academic  and  Behavioral  Competencies  Program 


Use  of  Materials  and  Strategies 

For  use  of  materials  and  strategies  to  support  SACD  goals,  87  comparisons  were  made,  with  4 expected  to  be 
significant  by  chance.  A significant  impact  was  found  on  treatment  teachers’  use  of  giveaways  in  Year  2 
(impact  = 31  percentage  points)  (table  2.14).  Regarding  use  of  instructional  techniques,  significantly  greater 
percentages  of  treatment  teachers  reported  using  journaling  in  Year  1 (impact  = 17  percentage  points), 
behavior  honor  roll  in  Year  2 (impact  = 32  percentage  points),  and  mentoring  in  Year  3 (impact  = 17 
percentage  points).  No  significant  impact  on  the  domain  of  materials  and  strategies  was  found  in  any  year. 
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Table  2.14.  Impacts  on  use  of  SACD  classroom  materials  and  teaching  strategies — ABC 


Year  1 

(Spring  3rd  grade) 

Year  2 

(Spring  4th  grade) 

Year  3 

(Spring  5th  grade) 

SACD  material  and  teaching  strategy 

T reat- 
ment 

Control 

Impact 

p-value 

T reat- 
ment 

Control 

Impact 

p-value 

T reat- 
ment 

Control 

Impact 

p-value 

Teacher  sample  size 

63 

73 

50 

67 

51 

64 

Use  of  SACD  materials  (percent) 

Teacher  guides  (manuals,  curricula) 

57.9 

64.4 

-6.5 

0.578 

57.7 

64.7 

-7.0 

0.528 

50.6 

47.8 

2.8 

0.796 

Student  materials  (workbooks  or  sheets) 

49.6 

63.1 

-13.5 

0.295 

57.3 

68.0 

-10.6 

0.261 

51.8 

51.3 

0.5 

0.957 

Instructional  aids  (games,  software,  videos) 

38.3 

43.6 

-5.3 

0.644 

37.0 

31.2 

5.8 

0.581 

26.9 

29.9 

-3.0 

0.768 

Giveaways  (bookmarks,  stickers) 

58.4 

51.9 

6.5 

0.571 

77.3* 

46.0 

31.2 

0.015 

61. 2A 

43.5 

17.7 

0.099 

Children’s  literature 

62.6 

47.0 

15.6 

0.138 

58.2 

50.4 

7.8 

0.518 

48.5 

38.8 

9.7 

0.420 

Other  types  of  materials 

6.5 

10.0 

-3.5 

0.563 

10.9 

11.4 

-0.5 

0.937 

10.4 

11.1 

-0.6 

0.924 

Did  not  use  any  of  these  materials 

7.7 

9.3 

-1.6 

0.818 

t 

t 

-7.7 

0.179 

13.5 

19.2 

-5.7 

0.336 

Use  of  teaching  strategies  (percent) 

Role-playing 

77.3 

76.3 

1.0 

0.941 

84.8 

74.1 

10.7 

0.316 

77.9 

71.5 

6.4 

0.489 

Cooperative  learning 

97.6 

96.6 

1.0 

0.761 

100.0 

97.9 

2.1 

t 

100.0 

98.9 

1.1 

t 

Peer  group  discussions 

85.5 

89.6 

-4.1 

0.513 

85.0 

89.9 

-4.9 

0.554 

94.9 

87.6 

7.3 

0.190 

Direct  instruction  of  SACD 

82.1 

78.9 

3.2 

0.755 

79.7 

67.5 

12.2 

0.471 

90.4 

81.7 

8.7 

0.287 

Skill  training 

44.9 

50.5 

-5.6 

0.685 

47.7 

37.4 

10.3 

0.395 

66.3 

59.7 

6.5 

0.437 

Incorporating  SACD  into  academic 
curriculum 

82.5 

79.7 

2.8 

0.776 

80.6 

71.6 

9.1 

0.368 

82.3 

80.6 

1.7 

0.848 

Parent  training 

7.1 

7.2 

-0.1 

0.977 

17.8 

8.3 

9.5 

0.186 

13.0 

12.8 

0.2 

0.984 

Parent  and  community  involvement 

20.6 

20.7 

-0.1 

0.988 

23.7 

10.6 

13.1 

0.150 

39.1 

33.9 

5.2 

0.546 

Mentoring 

39.1 

41.6 

-2.6 

0.814 

46.5 

39.9 

6.7 

0.518 

59.9* 

42.7 

17.2 

0.036 

Good  behavior  notes  sent  home  daily 
or  weekly 

96.2'' 

' 80.4 

15.9 

0.053 

91. 3A 

72.5 

18.8 

0.072 

94.4 

86.4 

8.0 

0.254 

Presenting  role  models 

74.3 

84.5 

-10.2 

0.344 

82. 0A 

70.2 

11.8 

0.066 

78.2 

78.3 

-0.1 

0.977 

See  notes  at  end  of  table. 


Chapter  2.  Academic  and  Behavioral  Competencies  Program 


194 


Table  2.14.  Impacts  on  use  of  SACD  classroom  materials  and  teaching  strategies — ABC — Continued 


Year  1 

Year  2 

Year  3 

(Spring  3rd  grade) 

(Spring  4th  grade) 

(Spring  5th  grade) 

SACD  material  and  teaching  strategy 

T reat- 
ment 

Control 

Impact 

p-value 

T reat- 
ment 

Control 

Impact 

p-value 

T reat- 
ment 

Control 

Impact 

p-value 

Use  of  teaching  strategies  (percent) — 
Continued 

Targeted  story  reading  or  writing  on  social 
and  character  development  themes 

80.4 

76.2 

4.3 

0.656 

79.2 

77.2 

2.0 

0.825 

87.7 

80.7 

7.0 

0.278 

Peer  mediation 

35.7 

47.8 

-12.1 

0.170 

52.9 ' 

' 34.5 

18.4 

0.071 

62.0 

51.7 

10.3 

0.278 

Honor  roll  for  positive  behavior 

71.2 

57.9 

13.3 

0.471 

76.0* 

43.8 

32.1 

0.037 

79.0 

65.9 

13.1 

0.259 

Pledges  or  recitations  on  social  and 
character  development  themes 

39.4 

56.3 

-16.9 

0.257 

34.9 

45.2 

-10.3 

0.542 

32.8 

61.3 

-28.5 

0.105 

Guided  visualization 

49.3 

53.6 

-4.3 

0.740 

49.1 

41.9 

7.1 

0.588 

51.8 

51.1 

0.7 

0.900 

Student-led/student-assisted  instruction 

60.2 

50.9 

9.3 

0.218 

53.4 

47.8 

5.6 

0.491 

65.7 

54.2 

11.5 

0.306 

Journaling 

90.4* 

73.4 

17.0 

0.014 

83.0 

65.6 

17.5 

0.176 

84.6 

76.7 

7.9 

0.290 

Time  out  for  negative  behavior 

78.4 

82.6 

-4.3 

0.725 

78.9 

73.1 

5.8 

0.458 

90.6 

82.9 

7.7 

0.408 

Daily  or  weekly  rewards  for  positive 
behavior 

97.1 

93.7 

3.3 

0.441 

95.0 

88.4 

6.6 

0.297 

100.0 

94.4 

5.6 

t 

Any  strategy 

100.0 

100.0 

t 

t 

100.0 

100.0 

0.0 

t 

100.0 

100.0 

0.0 

t 

Number  of  strategies  (mean) 

13.0 

12.9 

0.0 

0.980 

13.3 

11.5 

1.8 

0.163 

14.2 

13.4 

0.8 

0.258 

t Not  applicable 

t Reporting  standards  not  met.  Values  suppressed  to  protect  confidentiality. 

* Treatment  group  significantly  different  from  control  group  at  the  .05  level. 

A Treatment  group  significantly  different  from  control  group  at  the  .10  to  > .05  level. 

NOTE:  Weights,  which  assign  equal  weight  to  each  school  within  the  program,  were  used  in  producing  the  treatment,  control,  and  overall  means.  The  number  of  results  found 
significant  was  no  more  than  expected  by  chance. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 


Chapter  2.  Academic  and  Behavioral  Competencies  Program 


Chapter  2.  Academic  and  Behavioral  Competencies  Program 


Regarding  the  use  of  schoolwide  strategies,  18  comparisons  were  made  between  treatment  and  control 
teacher  reports,  with  1 expected  to  be  significant  by  chance.  In  Year  1,  a significantly  greater  percentage  of 
treatment  teachers  reported  that  their  schools  used  three  of  five  specific  schoolwide  strategies  (these  data  are 
not  shown  in  a table):  newspapers  or  bulletins  (impact  = 29  percentage  points),  special  school  days  (impact  = 
16  percentage  points),  and  special  events  (impact  = 24  percentage  points).  In  Years  2 and  3,  no  significant 
impacts  were  found  for  the  individual  schoolwide  strategies.  ABC  did  have  a significant  impact  on  the 
domain  of  schoolwide  strategies  in  Year  1. 

Participation  in  Professional  Development 

Regarding  reported  participation  in  professional  development,  27  comparisons  were  made  over  3 years,  with 
1 expected  to  be  significant  by  chance.  In  all  3 years,  the  intervention  had  no  statistically  significant  effects  on 
teachers’  participation  in  professional  development  for  activities  related  to  social  and  character  development 
overall  or  on  training  for  specific  SACD  goals  (table  2.15).  No  significant  impacts  were  found  on  the  domain 
in  all  3 years. 
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Table  2.15.  Impacts  on  teacher-reported  SACD  professional  development — ABC 


Year  1 Year  2 Year  3 

(Spring  3rd  grade) (Spring  4th  grade) (Spring  5th  grade) 

T reat-  T reat-  T reat- 


SACD  professional  development 

ment 

Control 

Impact 

p-value 

ment 

Control 

Impact 

p-value 

ment 

Control 

Impact 

p-value 

Teacher  sample  size 

63 

73 

50 

67 

51 

64 

SACD  training  in  past  12  months  (percent) 

77.3 

61.9 

15.3 

0.293 

51.4 

46.8 

4.6 

0.713 

40.4 

37.9 

2.5 

0.799 

Hours  of  SACD  training  (mean) 

9.8 

8.3 

1.4 

0.682 

7.4 

7.2 

0.2 

0.993 

2.5 

2.6 

-0.2 

0.854 

Training  by  goal  (percent) 

Violence  prevention  and  peace  promotion 

13.8 

27.7 

-13.9 

0.154 

17.9 

14.6 

3.2 

0.732 

13.1 

8.4 

4.7 

0.578 

Social  and  emotional  development 

23.6 

27.4 

-3.8 

0.751 

14.3 

14.6 

-0.4 

0.966 

t 

t 

-6.2 

0.251 

Character  education 

33.7 

39.3 

-5.6 

0.691 

16.2 

19.6 

-3.4 

0.694 

9.1 

14.3 

-5.2 

0.464 

Tolerance  and  diversity 

9.1 

19.4 

-10.3 

0.223 

13.3 

19.0 

-5.6 

0.505 

t 

i 

1.6 

0.749 

Risk  prevention  and  health  promotion 

6.5 

16.6 

-10.1 

0.110 

10.3 

5.5 

4.8 

0.321 

13.9 

4.3 

9.6 

0.164 

Civic  responsibility  and  community  service 

6.5 

4.4 

2.1 

0.679 

10.6 

6.1 

4.6 

0.440 

t 

i 

0.7 

0.794 

Behavior  management 

53.6 

31.6 

21.9 

0.211 

28.6 

25.6 

3.0 

0.829 

16.9 

20.0 

-3.2 

0.704 

4:  Reporting  standards  not  met.  Values  suppressed  to  protect  confidentiality. 

NOTE:  No  findings  were  found  statistically  significant  at  or  below  the  .05  level.  Weights,  which  assign  equal  weight  to  each  school  within  the  program,  were  used  in  producing  the 
treatment,  control,  and  overall  means. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 


Chapter  2.  Academic  and  Behavioral  Competencies  Program 


Chapter  2.  Academic  and  Behavioral  Competencies  Program 


Attitudes  and  Practices 

Teachers  reported  on  their  enthusiasm  for  SACD  efforts  in  their  schools  (these  data  are  not  shown  in  a table) 
by  indicating  enthusiasm,  cooperation,  or  open  dislike.  They  also  reported  on  the  SACD  practices  of  teachers 
and  staff  members  in  their  schools  (these  data  are  not  shown  in  a table).  These  practices  included  modeling 
positive  character  and  behavior  traits  with  students  and  fellow  teachers,  involving  students  in  making 
decisions,  giving  students  a voice  in  school  governance,  the  school  encouraging  parent  involvement  in 
children’s  social  and  character  development,  and  using  developmentally  appropriate  discipline  strategies 
rather  than  punishment  for  misbehavior.  In  each  year,  9 comparisons  were  made  between  treatment  and 
control  teachers  on  attitudes  and  practices  (27  total),  with  1 expected  to  be  significant  by  chance.  In  Year  1 
there  was  a statistically  significant  negative  impact  on  enthusiasm  (impact  = -5  percentage  points).  There  were 
significant  positive  impacts  on  encouraging  parent  involvement  in  Years  2 and  3 (impact  = 20  and  37 
percentage  points)  and  on  developmentally  appropriate  discipline  strategies  in  Year  2 (impact  = 26  percentage 
points).  ABC  had  a significant  positive  effect  on  the  domain  in  Year  3. 

Year-by-Year  Impacts  on  Students  and  Perceptions  of  School  Climate 

The  primary  research  question  for  the  ABC  evaluation  was  this: 

What  is  the  average  effect  of  the  ABC  program  on  children’s  social  and  emotional  competence, 
behavior,  academics,  and  perceptions  of  school  climate? 

The  first  approach  to  answering  this  question  was  to  examine  the  year-by-year  impacts  of  ABC  on  these 
student  and  school  climate  outcomes  over  the  3 years  as  the  students  progressed  from  third  through  fifth 
grades. 

Equation  (2)  (described  in  chapter  1)  was  estimated  to  provide  ABC’s  impacts  on  the  20  outcomes  based  on 
data  from  the  12  treatment  and  control  schools.  For  the  ABC  evaluation,  equation  (2)  excluded  the  program 
fixed  effects  (dp)  and  included  program-specific  covariates  and  random  school  effects  covariates.  Table  2.16 
lists  the  covariates  used  with  outcomes  from  each  report  in  the  ABC  analysis. 
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Table  2.16.  Covariates  used  with  outcomes  from  each  report  for  analysis — ABC 


Potential  covariate 

CR 

outcome 

PCR 

outcome 

TRS 

outcome 

TRCS 

outcome 

Total  number 

11 

13 

24 

8 

Child-reported 

Female 

Y 

Y 

Y 

Hispanic 

Y 

Y 

Y 

Black  (non-Hispanic) 

Y 

Y 

Y 

Other  ethnicity 

Y 

y 

Y 

Age  in  years 

>4 

Y 

Y 

Scales 

Afraid  at  School 

Y 

Altruistic  Behavior 

Y 

Y 

Empathy 

Engagement  with  Learning 

Y 

Negative  School  Orientation 

Y 

Normative  Beliefs  About  Aggression 

Sense  of  School  as  a Community 

Y 

Problem  Behavior 

Self-Efficacy  for  Peer  Interactions 

y 

Victimization  at  School 

Primary  caregiver-reported 

Age  in  years 

y 

Y 

Completed  high  school  or  equivalent 

Some  college 

Bachelor’s  or  higher  degree 

Highest  level  of  education  in  household 

Completed  high  school  or  equivalent 

Y 

Some  college 

Y 

Bachelor’s  or  higher  degree 

Y 

Mother  present  in  home  life 

y 

Y 

Mother  and  father  present 

Y 

Y 

Respondent  someone  other  than  mother  or  father 

Y 

Y 

Number  of  people  in  household 

Y 

Household  income:  $20,000  to  $40,000 

Household  income:  $40,000  to  $60,000 

Household  income:  More  than  $60,000 

Income-to-poverty-threshold  ratio:  Below  135  percent 

Income-to-poverty-threshold  ratio:  135  to  185  percent 

Full-time  employment 

Y 

Y 

Part-time  employment 

>4 

Y 

See  notes  at  end  of  table. 
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Table  2.16.  Covariates  used  with  outcomes  from  each  report  for  analysis — ABC — Continued 

CR  PCR  TRS  TRCS 


Potential  covariate  outcome  outcome  outcome  outcome 


Parental  scales 

APQ-Poor  Monitoring  and  Supervision  Subscale 

Y 

Y 

Y 

APQ-Positive  Parenting  Subscale 

Child-Centered  Social  Control 

Confusion,  Hubbub,  and  Order 

Y 

Y 

Community  Resources 

Community  Risk 

Parent  and  Teacher  Involvement 

Child  scales 

Altruistic  Behavior 

Positive  Social  Behavior 

Y 

Y 

Y 

Problem  Behavior 

Teacher-reported 

Female 

Y 

Hispanic 

Y 

Black  (non-Hispanic) 

Y 

Other  ethnicity 

Y 

Total  teaching  experience 

Y 

Total  experience  in  current  school 

Regular  certificate 

Y 

Other  certificate 

Y 

Highest  degree-bachelor’s 

Y 

Child  scales 

Academic  Competence  and  Motivation 

ADHD-Related  Behavior 

Altruistic  Behavior 

Positive  Social  Behavior 

Problem  Behavior 

Y 

Parent  and  Teacher  Involvement 

NOTE:  Abbreviations  are 
CR:  Child  Report 
PCR:  Primary  Caregiver  Report 
TRS:  Teacher  Report  on  Student 
TRCS:  Teacher  Report  on  Classroom  and  School 
ADHD:  Attention  deficit  hyperactivity  disorder 
APQ:  Alabama  Parenting  Questionnaire 
V : Covariate  used 
Blank  cell:  Covariate  not  used 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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To  assess  the  statistical  power  of  the  program -level  impact  estimates,  minimum  detectable  impacts  in  effect 
size  units  (MDES)  for  each  outcome  measure  were  calculated  for  the  ABC  evaluation  (table  2.17).  MDES 
represent  the  smallest  impacts  in  effect  size  (standard  deviation)  units  that  can  be  detected  with  a high 
probability  (80%).  MDES  are  primarily  a function  of  study  sample  sizes,  the  degrees  of  freedom  available  for 
statistical  tests,  and  design  effects  due  to  clustering  (Schochet  2005).  For  the  ABC  evaluation,  the  MDES 
ranged  from  0.110  to  0.551  for  the  child-level  outcomes  and  from  0.396  to  0.629  for  the  perception  of  school 
climate  outcomes  based  on  the  Teacher  Report  on  Classroom  and  School.  In  general,  the  MDES  for  the 
school  climate  outcomes  were  larger  than  those  for  the  child-level  outcomes. 
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Table  2.17.  Adjusted  minimum  detectable  effect  sizes  for  impact  evaluation — ABC 


Outcome  measure-Report 

Year  1 

Year  2 

Year  3 

Social  and  Emotional  Competence  Domain 

Self-Efficacy  for  Peer  Interaction-CR 

0.110 

0.130 

0.173 

Normative  Beliefs  About  Aggression-CR 

0.159 

0.159 

0.315 

Empathy-CR 

0.133 

0.225 

0.262 

Behavior  Domain 

Altruistic  Behavior-CR 

0.155 

0.110 

0.124 

Altruistic  Behavior-PCR 

0.189 

0.170 

0.236 

Altruistic  Behavior-TRS 

0.214 

0.551 

0.353 

Positive  Social  Behavior-PCR 

0.166 

0.148 

0.130 

Positive  Social  Behavior-TRS 

0.190 

0.423 

0.381 

Problem  Behavior-CR 

0.221 

0.184 

0.244 

Problem  Behavior-PCR 

0.231 

0.167 

0.143 

Problem  Behavior-TRS 

0.297 

0.295 

0.304 

ADHD-Related  Behavior-TRS 

0.297 

0.381 

0.402 

Academics  Domain 

Engagement  with  Learning-CR 

0.181 

0.110 

0.175 

Academic  Competence  and  Motivation-TRS 

0.149 

0.189 

0.320 

Perceptions  of  School  Climate  Domain 

Positive  School  Orientation-CR 

0.163 

0.311 

0.320 

Negative  School  Orientation-CR 

0.110 

0.189 

0.286 

Student  Afraid  at  School-CR 

0.242 

0.240 

0.243 

Victimization  at  School-CR 

0.156 

0.110 

0.211 

Feelings  of  Safety-TRCS 

0.491 

0.396 

0.586 

Student  Support  tor  Teachers-TRCS 

0.474 

0.474 

0.629 

NOTE:  Abbreviations  are 
CR:  Child  Report 
PCR:  Primary  Caregiver  Report 
TRS:  Teacher  Report  on  Student 
TRCS:  Teacher  Report  on  Classroom  and  School 
ADHD:  Attention  deficit  hyperactivity  disorder 
The  minimum  detectable  effect  (MDE)  formula  used  in  the  calculations  is  as  follows: 


MDE  = factored/ ) * 


Pi  (-  + -)  + (l-Pi)(— + — ) 
\sT  scJ  \sTnT  scncJ 


where  sT  and  sc  are  the  number  of  treatment  and  comparison  schools;  nT  and  nc  are  the  average  number  of  students  per 
classroom;  pi  is  the  intraclass  correlation  (ICC)  at  the  school  level;  and  factor(df)  is  a constant  that  depends  on  the  number  of 
degrees  of  freedom  (df)  available  for  analysis  (and  is  2.802  for  the  pooled  analysis). 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Table  2.18  provides  the  estimates  of  ABC’s  impacts  on  each  of  the  20  outcomes  over  each  of  the  3 years  (60 
comparisons  in  total,  with  3 expected  to  be  statistically  significant  by  chance).  Of  the  60  results,  5 were 
statistically  significant,  with  4 having  beneficial  impacts.  ABC  had  a significant  positive  impact  on  third- 
graders’  Altruistic  Behavior  (Teacher  Report  on  Student,  effect  size  [ES]  = 0.39);  on  fourth-graders’ 
Academic  Competence  and  Motivation  (Teacher  Report  on  Student,  ES  = 0.31)  and  Feelings  of  Safety 
(Teacher  Report  on  Classroom  and  School,  ES  = 0.75);  and  on  fifth-graders’  Positive  Social  Behavior 
(Primary  Caregiver  Report,  ES  = 0.21).  A negative  impact  was  found  on  fourth-graders’  Altruistic  Behavior 
(Child  Report,  ES  = —0.20).  ABC  had  two  beneficial  impacts  that  were  substantively  important  but  not 
statistically  significant:  (1)  on  fourth-graders’  Student  Support  for  Teachers  (Teacher  Report  on  Classroom 
and  School,  ES  = 0.27)  and  (2)  on  fifth-graders’  Feelings  of  Safety  (Teacher  Report  on  Classroom  and 
School,  ES  = 0.31).  Application  of  the  heuristics  to  adjust  for  multiple  comparisons  within  each  outcome 
domain  indicated  significant  beneficial  impacts  on  the  domains  of  Academics  and  Perceptions  of  School 
Climate  in  Year  2. 
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Table  2.18.  Impacts  on  child  and  school  outcomes — ABC 


Year  1 Year  2 Year  3 

(Spring  3rd  grade) (Spring  4th  grade) (Spring  5th  grade) 


Treat-  Effect  Treat-  Effect  Treat-  Effect 


Scale-Report 

ment 

Control 

size 

p-value 

ment 

Control 

size 

p-value 

ment 

Control 

size 

p-value 

Social  and  Emotional  Competence  Domain 

Self-Efficacy  for  Peer  Interactions-CR  (+) 

3.15 

3.11 

0.06 

0.464 

3.31 

3.29 

0.03 

0.770 

3.31 

3.23 

0.11 

0.315 

Normative  Beliefs  About  Aggression-CR  (-) 

1.35 

1.30 

0.09 

0.440 

1.39 

1.29 

0.18 

0.141 

1.51 

1.51 

0.00 

0.981 

Empathy-CR  (+) 

2.27 

2.26 

0.02 

0.823 

2.08 

2.19 

-0.21 

0.144 

2.00 

2.01 

-0.03 

0.859 

Behavior  Domain 

Altruistic  Behavior-CR  (+) 

1.29 

1.35 

-0.07 

0.449 

0.94* 

1.09 

-0.20 

0.029 

1.01 

0.98 

0.04 

0.656 

Altruistic  Behavior-PCR  (+) 

2.25 

2.15 

0.15 

0.249 

2.14 

2.19 

-0.06 

0.595 

2.28 

2.32 

-0.04 

0.767 

Altruistic  Behavior-TRS  (+) 

1.44* 

1.29 

0.39 

0.026 

1.24 

1.26 

-0.03 

0.914 

1.21 

1.22 

-0.05 

0.789 

Positive  Social  Behavior-PCR  (+) 

2.96 

2.96 

0.00 

0.993 

3.06A 

2.96 

0.16 

0.071 

3.06* 

2.94 

0.21 

0.041 

Positive  Social  Behavior-TRS  (+) 

2.90 

2.89 

0.02 

0.800 

3.01 

3.02 

-0.01 

0.962 

3.03 

3.16 

-0.19 

0.406 

Problem  Behavior-CR  (-) 

0.36 

0.34 

0.04 

0.713 

0.33 

0.36 

-0.05 

0.659 

0.51 

0.53 

-0.03 

0.812 

Problem  Behavior-PCR  (-) 

1.60 

1.60 

0.00 

0.976 

1.56 

1.61 

-0.11 

0.229 

1.58 

1.61 

-0.08 

0.465 

Problem  Behavior-TRS  (-) 

1.53 

1.52 

0.03 

0.850 

1.43 

1.46 

-0.07 

0.655 

1.54 

1.44 

0.21 

0.262 

ADHD-Related  Behavior-TRS  (-) 

1.84 

1.85 

-0.01 

0.958 

1.69 

1.77 

-0.13 

0.525 

1.72 

1.62 

0.17 

0.447 

Academics  Domain1 

Engagement  with  Learning-CR  (+) 

3.61 

3.61 

0.00 

0.986 

3.68 

3.63 

0.10 

0.281 

3.58 

3.58 

0.00 

0.964 

Academic  Competence  and  Motivation- 

TRS  (+) 

2.81 

2.79 

0.03 

0.719 

2.92* 

2.65 

0.31 

0.011 

2.76 

2.77 

-0.02 

0.924 

See  notes  at  end  of  table. 
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Table  2.18.  Impacts  on  child  and  school  outcomes — ABC — Continued 


Year  1 

(Spring  3rd  grade) 

Year  2 

(Spring  4th  grade) 

Year  3 

(Spring  5th  grade) 

Scale-Report 

T reat- 
ment 

Control 

Effect 
size  i 

o-value 

T reat- 
ment 

Control 

Effect 

size 

p-value 

T reat- 
ment 

Control 

Effect 

size 

p-value 

Perceptions  of  School  Climate  Domain2 
Positive  School  Orientation-CR  (+) 

2.82 

2.94 

-0.17 

0.143 

2.67 

2.61 

0.07 

0.668 

2.41 

2.45 

-0.05 

0.773 

Negative  School  Orientation-CR  (-) 

1.94 

1.85 

0.16 

0.108 

1.97 

1.96 

0.01 

0.963 

2.16 

2.12 

0.07 

0.680 

Student  Afraid  at  School-CR  (-) 

2.23 

2.28 

-0.06 

0.707 

2.21 

2.36 

-0.17 

0.263 

2.19 

2.35 

-0.19 

0.218 

Victimization  at  School-CR  (-) 

0.74 

0.81 

-0.08 

0.417 

0.58 

0.66 

-0.11 

0.213 

0.66 

0.82 

-0.19 

0.151 

Feelings  of  Safety-TRCS  (+) 

3.42 

3.25 

0.21 

0.353 

3.59* 

2.97 

0.75 

0.003 

3.29 

3.03 

0.31° 

0.235 

Student  Support  for  Teachers-TRCS  (+) 

3.27 

3.28 

0.00 

0.979 

3.48 

3.26 

0.27° 

0.276 

3.25 

3.21 

0.05 

0.847 

* Treatment  group  significantly  different  from  control  group  at  the  .05  level. 

A Treatment  group  significantly  different  from  control  group  at  the  .10  to  > .05  level. 

° Substantive  (but  nonsignificant  at  .05  level)  effect  size  of  s .25  or  < -.25. 

1 1n  Year  2,  at  least  half  of  the  impacts  were  positive  and  statistically  significant  and  no  impact  was  negative  and  statistically  significant  based  on  univariate  statistical  tests.  In  addition, 
at  least  one  outcome  remained  positive  and  statistically  significant  and  no  outcome  was  negative  and  statistically  significant  after  applying  the  Benjamini-Hochberg  (1995)  procedure  to 
adjust  significance  levels  downward  to  account  for  the  multiple  testing  of  impacts. 

2 In  Year  2,  at  least  one  outcome  remained  positive  and  statistically  significant  and  no  outcome  was  negative  and  statistically  significant  after  applying  the  Benjamini-Hochberg  (1995) 
procedure  to  adjust  significance  levels  downward  to  account  for  the  multiple  testing  of  impacts. 

NOTE:  Abbreviations  are 
CR:  Child  Report 
PCR:  Primary  Caregiver  Report 
TRS:  Teacher  Report  on  Student 
TRCS:  Teacher  Report  on  Classroom  and  School 
ADHD:  Attention  deficit  hyperactivity  disorder 

The  +/-  signs  in  parentheses  indicate  the  direction  of  a beneficial  outcome.  All  Impact  estimates  were  calculated  using  regression  models,  where  each  program  and  school  within  a 
program  was  weighted  equally.  The  standard  errors  of  all  estimates  account  for  design  effects  due  to  unequal  weighting  and  the  clustering  of  students  within  schools.  The  effect  size 
was  calculated  by  dividing  the  estimated  Impact  by  the  standard  deviation  of  the  outcome  measure  for  the  control  group.  See  table  1.5  for  information  about  the  measures  used  to 
create  the  outcome  variables. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Impacts  on  Child  Outcomes  Over  Time 

ABC  impacts  on  the  child  outcomes  over  time  were  estimated  using  growth  curve  models  by  examining 
treatment  and  control  group  differences  in  the  trajectories  of  student  outcomes  during  the  follow-up  period 
while  accounting  for  clustering  at  the  school  level.  The  growth  curve  models  are  estimated  using  a three-level 
hierarchical  linear  model,  where  Level  1 corresponds  to  time,  Level  2 to  students,  and  Level  3 to  schools 
(described  in  chapter  1). 

Table  2.19  provides  the  estimates  of  the  ABC  impacts  on  the  growth  in  student  outcomes  over  the  3 years. 
The  estimated  impacts  range  in  effect  size  units  (absolute  value)  from  0.01  to  0.12.  ABC  did  not  have  a 
statistically  significant  effect  on  any  of  the  18  estimated  impacts  (1  is  expected  to  be  significant  by  chance). 
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Table  2.19.  Impacts  on  growth  of  child  outcomes — ABC 


Scale-Report 

Average  growth  in  the  score  per  year1 

Mean  score  at 

implementation2  Treatment 

Control 

Impact  on 
growth3 

Effect 

4 

size 

Standard 
error  of 
impact 

p-value  of 
impact 

Social  and  Emotional  Competence  Domain 

Self-Efficacy  for  Peer  Interactions-CR  (+) 

3.05 

0.13 

0.07 

0.06 

0.07 

0.03 

0.116 

Normative  Beliefs  About  Aggression-CR  (-) 

1.28 

0.1 0A 

0.02 

0.08 

0.12 

0.04 

0.078 

Empathy-CR  (+) 

2.35 

-0.1 5A 

-0.10 

-0.05 

-0.11 

0.03 

0.096 

Behavior  Domain 

Altruistic  Behavior-CR  (+) 

1.47 

-0.20 

-0.21 

0.01 

0.01 

0.05 

0.908 

Altruistic  Behavior-PCR  (+) 

2.29 

-0.04 

-0.02 

-0.02 

-0.02 

0.04 

0.654 

Altruistic  Behavior-TRS  (+) 

1.33 

-0.06 

-0.01 

-0.05 

-0.10 

0.05 

0.325 

Positive  Social  Behavior-PCR  (+) 

2.91 

0.05A 

0.00 

0.05 

0.08 

0.03 

0.097 

Positive  Social  Behavior-TRS  (+) 

2.89 

0.05 

0.11 

-0.07 

-0.08 

0.06 

0.293 

Problem  Behavior-CR  (-) 

0.29 

0.08 

0.06 

0.02 

0.04 

0.03 

0.451 

Problem  Behavior-PCR  (-) 

1.63 

-0.01 

0.00 

-0.01 

-0.02 

0.02 

0.611 

Problem  Behavior-TRS  (-) 

1.45 

0.05A 

-0.01 

0.06 

0.11 

0.03 

0.057 

ADHD-Related  Behavior-TRS  (-) 

1.84 

-0.06 

-0.05 

0.00 

-0.01 

0.06 

0.935 

Academics  Domain 

Engagement  with  Learning-CR  (+) 

3.64 

-0.02 

-0.01 

-0.01 

-0.01 

0.04 

0.835 

Academic  Competence  and  Motivation-TRS  (+) 

2.70 

0.02 

0.01 

0.01 

0.01 

0.05 

0.858 

See  notes  at  end  of  table. 

Chapter  2.  Academic  and  Behavioral  Competencies  Program 


207 


Table  2.19.  Impacts  on  growth  of  child  outcomes — ABC — Continued 


Average  growth  in  the  score  per  year1 


Scale-Report 

Mean  score  at 
implementation2 

Treatment 

Control 

Impact  on 
growth3 

Effect 

size4 

Standard 
error  of 
impact 

p-value  of 
impact 

Perceptions  of  School  Climate  Domain 
Positive  School  Orientation-CR  (+) 

3.12 

-0.27 

-0.22 

-0.05 

-0.06 

0.05 

0.343 

Negative  School  Orientation-CR  (-) 

1.85 

0.09 

0.07 

0.02 

0.02 

0.05 

0.691 

Student  Afraid  at  School-CR  (-) 

2.29 

-0.02 

0.02 

-0.04 

-0.04 

0.06 

0.484 

Victimization  at  School-CR  (-) 

0.74 

-0.03 

0.01 

-0.05 

-0.05 

0.04 

0.248 

A Treatment  group  significantly  different  from  control  group  at  the  .10  to  > .05  level. 

1 Pertains  to  the  estimated  slope  of  the  outcome  for  the  treatment  or  control  groups. 

2 The  average  score  at  implementation  is  calculated  across  treatment  and  control  groups,  using  regression  models  for  adjustment  on  covariates. 

3 Estimated  difference  between  the  slope  of  the  treatment  and  control  groups. 

4 Effect  size:  the  slope  of  the  treatment  group  minus  the  slope  of  the  control  group  divided  by  the  standard  deviation  of  the  outcome  for  the  program's  control  group  (the  standard 
deviation  is  calculated  without  accounting  for  school-level  clustering  or  regression  adjustments). 

NOTE:  Abbreviations  are 
CR:  Child  Report 
PCR:  Primary  Caregiver  Report 
TRS:  Teacher  Report  on  Student 
ADHD:  Attention  deficit  hyperactivity  disorder 

No  findings  were  found  statistically  significant  at  or  below  the  .05  level.  The  +/-  signs  in  parentheses  indicate  the  direction  of  a beneficial  outcome.  All  Impact  estimates  were  calculated 
using  HLM  6.06.  Sample  weights  were  used  in  all  analyses  to  give  (1)  each  school  equal  weight  in  each  program  (within  each  time  period)  and  (2)  each  time  period  equal  weight  within 
the  analysis.  See  table  1 .5  for  information  about  the  measures  used  to  create  the  outcome  variables. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 


Chapter  2.  Academic  and  Behavioral  Competencies  Program 


Chapter  2.  Academic  and  Behavioral  Competencies  Program 


Summary 

As  part  of  the  Social  and  Character  Development  (SACD)  initiative,  researchers  at  the  Buffalo  site 
implemented  and  evaluated  the  ABC  program,  which  consists  of  multiple  components  related  to  classroom 
behavior  management  and  teaching  and  promoting  social  skills.  Twelve  public  schools  in  three  school 
districts  in  upstate  New  York  were  recruited  by  the  Buffalo  research  team  and  randomly  assigned  to 
treatment  and  control  conditions  to  determine  the  impact  of  ABC  on  social  and  character  development 
activities  in  the  schools  and  on  the  child  outcome  domains  of  Social  and  Emotional  Competence,  Behavior, 
Academics,  and  Perceptions  of  School  Climate. 

Analyses  of  the  initial  characteristics  of  the  sample  (students,  caregivers,  communities,  teachers,  and  schools) 
indicated  that  randomization  to  treatment  and  control  status  produced  groups  that  were  similar  on  the 
observed  characteristics  near  the  start  of  the  study  (none  of  the  comparisons  were  statistically  significantly 
different).  Documentation  of  the  initial  level  of  social  and  character  development  activities  in  the  schools 
revealed  six  differences  between  the  treatment  and  control  teachers  and  classrooms,  more  than  would  be 
expected  by  chance. 

The  data  on  the  initial  level  of  SACD  activity  led  to  two  findings.  First,  treatment  teachers  reported  greater 
use  of  and  training  in  SACD  activities  than  did  control  teachers,  and  they  did  so  more  often  than  would  be 
expected  by  chance.  There  are  two  potential  causes  for  this  finding,  and  the  analysis  cannot  be  used  to 
determine  whether  the  reason  for  such  a difference  was  that  the  two  groups  did  differ  on  their  initial  use  of 
SACD  activities  (i.e.,  that  randomization  did  not  create  similar  treatment  and  control  groups)  or  the  fact  that 
the  training  of  all  treatment  teachers  and  the  implementation  of  the  program  before  the  initial  data  were 
collected  influenced  the  teacher  reports  (program  implementation  began  at  the  treatment  schools  on  average 
6 weeks  before  the  fall  2004  data  collection  occurred).  Because  it  is  likely  (though  unproven)  that  the  training 
and  implementation  affected  the  teacher  reports,  these  data  were  not  considered  appropriate  for  use  as  a 
baseline  measure  of  SACD  activities  and  training  in  the  treatment  schools. 

Second,  these  data  indicated  that  the  control  condition  for  the  SACD  project  was  not  a “no  treatment” 
control  but  a “standard  practice”  control.  Because  the  control  teachers  were  not  affected  by  the 
implementation  of  the  SACD  programs  before  data  collection,  their  reports  reflect  standard  practice  in  the 
control  schools.  Standard  practice  at  the  control  schools  included  reports  of  55  percent  to  88  percent  of 
teachers  using  SACD  activities,  88  percent  of  teachers  reporting  the  use  of  specific  materials  in  conjunction 
with  these  activities,  1 00  percent  reporting  the  use  of  at  least  one  of  the  specified  instructional  strategies,  and 
71  percent  reporting  participation  in  SACD  training  over  the  past  12  months. 

Analyses  of  ABC  impacts  on  the  use  of  SACD  activities  in  the  schools  revealed  few  differences  between 
treatment  and  control  schools.  Five  of  the  90  program  impacts  on  SACD  activities,  4 of  the  87  impacts  on 
SACD  materials  and  strategies,  3 of  the  18  impacts  on  schoolwide  strategies,  and  4 of  the  27  impacts  on 
attitudes  and  practices  (1  negative)  were  statistically  significant.  No  impacts  on  professional  development 
were  found  in  any  of  the  years. 

Of  the  20  childdevel  outcomes  representing  the  four  domains  of  Social  and  Emotional  Competence, 
Behavior,  Academics,  and  Perceptions  of  School  Climate  assessed  in  each  of  the  3 years  of  the  study  (a  total 
of  60  results),  5 were  statistically  significant.  One  of  these  5 was  a detrimental  impact  and  4 were  beneficial 
impacts.  There  were  no  effects  of  the  ABC  program  over  time  as  assessed  using  growth  curve  models. 

The  SACD  evaluation  did  not  find  evidence  to  support  the  hypothesis  that  the  ABC  program  had  beneficial 
impacts  on  students’  social  and  character  development.  These  results  could  have  been  due  to  the  inability  of 
the  program  to  cause  such  change,  possibly  because  the  theory  of  action  was  incomplete  or  because  the 
activities  to  carry  out  that  theory  were  not  effective. 
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These  results  may  also  be  due  to  the  inability  of  the  evaluation  to  observe  such  an  impact  due  to  the  control 
condition,  the  level  of  nonparticipation,  or  the  sample  size.  The  control  schools  continued  using  their 
standard  SACD  activities,  and  these  turned  out  to  be  high  in  quantity  and  broad  in  scope.  New  York  passed 
the  Safe  Schools  Against  Violence  in  Education  Act  of  2001  to  encourage  districts  to  promote  instruction  in 
civility,  citizenship,  and  character  education,  and  this  may  have  been  a factor  in  high  SACD  activity  in  the 
control  schools.  The  high  level  of  control  school  activity,  combined  with  relatively  few  significant  impacts  by 
the  ABC  program  on  the  use  of  SACD  activities  in  the  treatment  schools,  may  have  caused  the  resulting 
difference  in  the  amount  of  SACD  activities  between  the  treatment  and  control  schools  to  be  too  small  to 
cause  significant  differences  in  the  student  outcomes.  In  addition,  about  one-third  of  the  students  in  the 
sample  universe  did  not  take  part  because  of  nonconsent  or  noncompletion  of  the  surveys.  As  a 
determination  could  not  be  made  as  to  whether  these  students  significantly  differed  from  those  who  did  take 
part,  the  evaluation’s  results  are  valid  only  for  the  students  who  took  part.  If  the  students  not  taking  part  were 
different,  and  if  they  would  have  responded  better  to  the  ABC  program  than  to  the  SACD  activities  occurring 
in  the  control  schools,  then  the  evaluation  could  have  underestimated  the  program’s  impact.  Also,  the  sample 
size  of  12  schools  and  the  resulting  higher  MDES  compared  to  those  for  the  multiprogram  evaluation  may 
have  reduced  the  likelihood  of  detecting  statistically  significant  effects.  However,  it  should  be  noted  that  58 
percent  of  the  MDES  for  the  60  outcomes  used  in  the  year-by-year  analysis  were  below  0.25  (47%  were 
below  0.20).  In  addition,  only  2 of  the  60  outcomes  were  found  to  be  substantively  important. 
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University  of  North  Carolina  at  Chapel  Hill 
(North  Carolina  Site) 


Intervention 

Researchers  at  the  University  of  North  Carolina  at  Chapel  Hill  (North  Carolina  site)  implemented  and 
evaluated  the  Cotnpetence  Support  Program  (CSP).  The  program,  developed  at  the  School  of  Social  Work  at  the 
University  of  North  Carolina  at  Chapel  Hill,  is  a combination  of  three  programs:  the  Making  Chokes  program 
(Fraser  et  al.  2000),  the  Competence  Enhancement  Behavior  Management  program  (Farmer  et  al.  2006),  and  the  Social 
Dynamics  Training  program  (Farmer  2000).  It  is  a universal,  classroom-level  intervention  that  functions  as  an 
add-on  curriculum  made  up  of  scripted  lessons  and  training  for  teachers  in  behavior  management  and  social 
dynamics.  Table  3.1  describes  CSP’s  general  characteristics  (panel  1),  the  types  of  instruction  and  strategies 
used  (panel  2),  the  professional  development  provided  for  those  implementing  the  program  (panel  3),  and  the 
social  and  character  development  activities  (panel  4)  and  outcomes  (panel  5)  addressed  by  the  program. 

• In  the  third  grade,  the  Making  Choices  curriculum  consists  of  28  scripted  lessons  that  focus  on  social 
problem  solving  and  involve  learning  about  emotions  and  feelings,  identifying  social  cues,  making 
sense  of  social  cues,  setting  social  goals,  choosing  options,  making  choices,  and  acting  on  choices. 
The  third  grade  is  considered  a keystone  year.  In  the  first  and  second  grades,  the  Making  Choices 
curriculum  includes  eight  scripted  lessons  per  year  that  prepare  children  for  the  third -grade  program 
and  focus  on  emotions  and  simple  emotion  regulation  techniques  (e.g.,  stop  and  think).  In  the  fourth 
and  fifth  grades,  the  program  consists  of  eight  scripted  lessons  per  year  that  focus  on  the  application 
of  problem-solving  skills  to  routine  peer  situations  (e.g.,  social  aggression  and  bullying).  Across  all 
grades,  teachers  engage  students  by  using  a variety  of  learning  strategies — story  reading,  discussion, 
role-playing,  worksheets,  art  projects,  and  games.  Lessons  are  designed  to  last  approximately  45 
minutes. 

• The  Contpetence  Enhancement  Behavior  Management  component  is  an  in-service  training  and  direct 
consultation  program  for  teachers  and  is  developed  from  evidenced-based  behavioral  and  ecological 
best  practices  for  promoting  positive  classroom  behavior.  It  includes  training  in  establishing 
proactive  behavior  management  aims  and  goals,  establishing  productive  classroom  routines  and 
structures,  teaching  and  reinforcing  alternative  behaviors,  building  supportive  relationships, 
communicating  with  troubled  students,  using  constructive  discipline  and  natural  consequences,  and 
preventing  and  managing  behavioral  crises. 

• Social  Dynamics  Training  is  an  in-service  teacher  training  and  consultation  model  that  includes  training 
teachers  to  recognize  the  influence  of  peer  social  structures  on  students’  behavior  and  to  manage 
classroom  social  dynamics.  Teachers  are  trained  to  monitor  and  intervene  in  processes  of  social 
aggression,  prevent  the  establishment  of  social  hierarchies  that  promote  jealousy  and  conflict, 
prevent  social  roles  that  promote  aggression  and  violence  (e.g.,  bullies),  and  monitor  and  prevent  the 
establishment  of  antisocial  and  enemy  peer  groups. 
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Table  3.1 . Competence  Support  Program 

Panel  1 : General  characteristics 

Target  population 
Universal 

Program  components 

Peer:  In  and  out  of  class 
Parent'.  None 

Classroom:  Lessons,  behavior  management,  and  social  dynamics 
Schoolwide:  None  or  not  major  focus 
Community:  None  or  not  major  focus 

Training:  Pretraining  and  ongoing  for  teachers  and  counselors 

Level  of  integration 

Add-on  curriculum 

Flexibility 

Manualized:  Scripted  lessons  for  teachers 

Adaptability:  Teachers  may  adapt  program  to  setting 

See  notes  at  end  of  table. 


Panel  2:  Description  of  instruction  and  strategies 

Classroom 

Lessons 

Who  delivers:  T eacher 

Activities  and  tools:  Story  reading,  discussion,  role-playing,  worksheets,  games,  artifacts  (e.g.,  turtle 
puppet),  art  projects 

Content:  Problem  solving,  emotion  identification  and  regulation,  identifying  social  cues,  goal  formation, 
choosing  options,  behavior  regulation 
Frequency:  45  minutes,  once  per  week,  28  lessons 
Strategies 

Who  delivers:  T eacher 

Activities  and  tools:  Prevention  of  social  hierarchies  that  promote  conflict,  social  reinforcement  for 
positive  behavior 
Frequency:  Daily 

Supplement  to  classroom 
None 

Schoolwide  activities 

None 

See  notes  at  end  of  table. 
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Table  3.1 . Competence  Support  Program — Continued 

Panel  3:  Professional  development 

Pre-implementation 

Teachers 

Content:  Training  on  curriculum  implementation,  behavior  management,  and  social  dynamics 
Duration:  4 hours 
Other 

Content:  Principal  consultation  on  development  of  schoolwide  management  policy 
Duration:  As  requested 

Ongoing  consultation 
Teachers 

Content:  Consultation  on  use  of  program  materials 
Duration:  2 times  per  month 
Other 

Content:  None 

Duration:  None 

See  notes  at  end  of  table. 


Panel  4:  Activities  for  SACD  goals 


Violence  prevention  and  peace  promotion  V 

Social  and  emotional  development  v 

Character  education 


Risk  prevention  and  health  promotion 
Civic  responsibility  and  community  service 
Behavior  management 


✓ 


Tolerance  and  diversity 
See  notes  at  end  of  table. 


Panel  5:  SACD  outcomes  addressed 


Engagement  with  Learning 

✓ 

Empathy 

✓ 

Academic  Competence  and  Motivation 

Positive  School  Orientation 

Altruistic  Behavior 

Negative  School  Orientation 

Positive  Social  Behavior 

✓ 

Student  Afraid  at  School 

Problem  Behavior 

✓ 

Victimization  at  School 

✓ 

Self-Efficacy  for  Peer  Interactions 

✓ 

Feelings  of  Safety 

Normative  Beliefs  About  Aggression 

✓ 

Student  Support  for  Teachers 

✓ 

NOTE:  Abbreviations  are 

■S : Activity  or  outcome  addressed 

Blank  cell:  Activity  or  outcome  not  addressed 


SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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• Teachers  receive  4 hours  of  training  on  curriculum  implementation,  behavior  management  strategies, 
and  social  dynamics.  Principals  receive  consultation  on  the  development  of  schoolwide  management 
policy,  as  requested.  Teachers  receive  consultations  on  the  use  of  program  materials  twice  a month. 

• Consultation  to  school  staff  is  a standard  element  of  CSP.  This  consultation  is  provided  by  school 
personnel,  such  as  guidance  counselors,  master  teachers,  school  psychologists,  or  school  social 
workers,  who  are  trained  to  be  program  specialists. 

Sample  and  Random  Assignment 

The  North  Carolina  research  team  recruited  a total  of  10  public  elementary  schools  in  2 school  districts  in  2 
counties  in  North  Carolina.  The  10  schools  were  randomly  assigned  to  treatment  and  control  conditions  prior 
to  the  fall  2004  data  collection.47  A two-step  process  was  used.  First,  a computer-generated  pairwise  matching 
algorithm  developed  by  Mathematica  Policy  Research,  Inc.  (MPR)  was  used  to  identify  the  best  pairwise 
matches  across  the  10  schools  based  on  variables  identified  by  the  North  Carolina  research  team.  The 
variables  used  in  the  pairwise  matching  for  North  Carolina  included  (a)  total  number  of  students; 
(b)  percentages  of  students  eligible  for  free  and  reduced-price  lunch;  (c)  percentages  of  students  who  were 
White/Black/Hispanic/Other;  (d)  math  achievement  scores;  and  (e)  reading  achievement  scores.  Then,  using 
the  flip  of  a coin,  1 school  in  each  matched  pair  was  assigned  to  either  the  intervention  or  control  condition, 
resulting  in  5 schools  receiving  CSP  and  5 schools  acting  as  control  schools  and  continuing  to  implement  the 
social  and  character  development  activities  that  constituted  their  standard  practice.  Assignment  to  treatment 
or  control  condition  was  at  the  school  level  and  therefore  limited  the  risk  of  contamination  between 
treatment  and  control  classrooms. 

All  10  schools  remained  in  the  study  for  the  full  3 years;  however,  one  K-5  school  in  the  treatment  group 
became  a K-4  school  and  continued  implementation  of  the  intervention.  The  cohort  of  students  from  that 
school  who  were  in  fifth  grade  during  the  2006-07  school  year  went  to  an  intermediate  school  that  had  fifth 
and  sixth  grades.  The  intermediate  school  did  not  implement  the  intervention.  These  fifth-grade  students  who 
were  part  of  the  original  cohort  were  followed  to  their  intermediate  school,  and  they,  their  primary  caregivers, 
and  their  fifth-grade  teachers  at  the  intermediate  school  provided  data  that  were  used  in  all  analyses  of  the 
impact  of  CSP  on  children  and  on  the  use  of  activities  related  to  social  and  character  development  in  the 
schools.  This  change  in  grade  configuration,  which  resulted  in  the  absence  of  the  intervention  for  that 
subgroup  of  fifth-graders  and  their  teachers,  should  be  noted  when  considering  the  results  for  the  North 
Carolina  site. 

The  original  student  sample  (the  cohort  of  students  in  the  third  grade  in  the  10  schools  in  fall  2004) 
numbered  959  students  (476  treatment  and  483  control).  Table  3.2  documents  the  change  in  the  sample  over 
the  3 spring  follow-up  data  collection  periods.  Over  time,  new  entrants  to  the  cohort  became  a larger 
percentage  of  the  sample,  eventually  making  up  24  percent  of  the  sample  by  the  spring  of  Year  3.  In  Year  2 
there  was  a statistically  significant  difference  between  the  treatment  and  control  groups  in  the  number  of  new 
entrants  (in  this  case,  fewer  in  the  control  group).  The  percentage  of  the  sample  made  up  of  the  original 
cohort  further  declined  as  students  left  the  schools.  By  Year  3,  approximately  25  percent  of  the  original 
sample  had  left.  In  Years  2 and  3 there  was  a statistically  significant  difference  between  treatment  and  control 
groups,  with  a larger  percentage  of  leavers  in  the  treatment  group. 


47  In  Year  2 (fall  2005),  the  North  Carolina  research  team  recruited  four  more  schools  from  an  additional  county,  two 
assigned  to  treatment  and  two  assigned  to  control,  which  were  followed  to  the  end  of  the  study  (spring  2007).  A 
description  of  this  second  cohort  and  all  relevant  findings  can  be  found  in  appendix  A.  The  data  from  this  second 
cohort  are  not  included  in  the  analyses  and  results  reported  in  this  chapter. 
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Table  3.2.  Sample — CSP 


Year  1 Year  1 Year  2 Year  3 

(Fall  3rd  grade) (Spring  3rd  grade)  (Spring  4th  grade)  (Spring  5th  grade) 


Treat-  Treat-  Treat-  Treat- 


Characteristic 

Total 

ment 

Control 

Total 

ment 

Control 

Total 

ment 

Control 

Total 

ment 

Control 

School  sample  size 

10 

5 

5 

10 

5 

5 

10 

5 

5 

101 

5 

5 

Student  sample  size 

959 

476 

483 

975 

485 

490 

969 

474 

495 

947 

458 

489 

Stayers 

t 

t 

t 

923 

456 

467 

729 

341 

388 

721 

337 

384 

New  entrants 

t 

t 

t 

52 

29 

23 

240 

133 

107 

226 

121 

105 

New  entrants  as  a percent  of 
spring  enrollment 

t 

t 

t 

5.3 

6.0 

4.7 

24.8 

28.1* 

21.6 

23.9 

26.4 

21.5 

Total  leavers  (from  original  cohort) 

t 

t 

t 

36 

20 

16 

230 

135 

95 

238 

139 

99 

Leavers  as  a percent  of  fall  2004 
enrollment 

t 

t 

t 

3.8 

4.2 

3.3 

24.0 

28.4** 

19.7 

24.8 

29.2** 

20.5 

Number  of  students  per  school 

(mean) 

96 

95 

97 

98 

97 

98 

97 

95 

99 

95 

92 

98 

Range  of  number  of  students  per 

school 

46-148 

85-106 

46-148 

47-155 

86-117 

47-155 

46-144 

84-121 

46-144 

47-139 

73-105 

47-139 

t Not  applicable. 

* Treatment  group  significantly  different  from  control  group  at  the  .05  level. 

**  Treatment  group  significantly  different  from  control  group  at  the  .01  level. 

1 One  of  the  treatment  group  schools  that  had  been  a K-5  school  in  prior  years  became  a K-4  school  and  continued  implementation  of  the  intervention.  The  Cohort  1 students,  who 
were  in  fifth  grade  during  the  2006-07  school  year,  went  to  an  intermediate  school  that  had  fifth  and  sixth  grades.  The  intermediate  school  did  not  implement  the  intervention.  These 
fifth-grade  students,  their  primary  caregivers,  and  their  teachers  were  included  in  the  spring  2007  data  collection  and  all  analyses  based  upon  spring  2007  data. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Implementation 

Training 

The  intervention  teachers  received  4 hours  of  program  implementation  training  prior  to  or  within  the  first  2 
weeks  of  the  beginning  of  the  school  year  (see  panel  3 of  table  3.1).  Teachers  had  access  to  ongoing  program 
implementation  support  throughout  the  school  year  from  two  consultants.  One  consultant  was  a program 
specialist  who  focused  on  the  Making  Choices  Program.  She  provided  onsite  consultation  in  the  form  of 
biweekly  planning  meetings  with  teachers  to  go  over  lesson  content  and  to  discuss  any  implementation  issues. 
The  second  consultant,  a program  specialist  for  the  Competence  Enhancement  and  Social  Dynamics  Training 
components,  conducted  biweekly  consultation  meetings  with  teacher  teams  to  discuss  key  concepts  in 
program  implementation  and  issues  and  strategies  around  specific  case  examples  generated  by  the  teachers.  In 
addition,  each  program  specialist  met  at  least  once  per  semester  with  each  principal  to  report  on  program 
activities.  These  two  consultants  also  consulted  with  principals  on  the  development  of  schoolwide 
management  policy  on  an  as-needed  basis. 

Data  Collection 

MPR  collected  the  child,  teacher,  and  school  data  at  the  North  Carolina  site.  Table  3.3  presents  the  data 
collection  dates.  Data  were  collected  in  the  fall  and  spring  of  the  first  2 years  and  the  spring  of  Year  3.  The 
fall  2004  data  collection  began  on  August  30,  2004,  and  ended  on  September  16,  2004.  The  average  time 
frame  from  the  beginning  of  program  implementation  to  the  beginning  of  fall  data  collection  was  4 weeks.  As 
a result,  initial  data  collection  took  place  after  CSP  implementation  began.  Therefore,  these  data  provide  a 
measure  of  the  students,  teachers,  and  schools  near  the  beginning  of  the  school  year,  at  a time  when  CSP  had 
been  operating  for  a relatively  short  period  of  time.  The  spring  data  collection  window  was  from  April  1 1 , 
2005,  to  April  29,  2005.  CSP  had  been  implemented  for  36  weeks  at  the  time  of  the  spring  data  collection  and 
for  30  weeks  from  the  end  of  the  fall  data  collection.  Year  2 followed  a similar  pattern,  with  implementation 
occurring  at  the  start  of  the  school  year,  fall  data  collection  occurring  5 weeks  later,  and  spring  data  collection 
occurring  27  weeks  after  fall  data  collection  (and  35  weeks  after  the  start  of  implementation).  In  spring  2007, 
data  collection  occurred  35  weeks  after  the  start  of  implementation.  During  the  2005-06  and  2006-07  school 
years,  one  school  operated  on  a year-round  schedule.  Data  collection  took  from  3 to  5 weeks  at  each 
collection  point. 
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Table  3.3.  Data  collection  dates — CSP 


Year  1 Year  1 Year  2 Year  2 Year  3 

Data  collection  schedule  (Fall  3rd  grade)  (Spring  3rd  grade)  (Fall  4th  grade)  (Spring  4th  grade)  (Spring  5th  grade) 


School  sample  size 

10 

10 

10 

10 

10 

School  year  dates 
First  day  of  school 

7/26/04 

8/2/041 

t 

8/25/05" 

t 

8/25/06" 

Start  of  implementation 

8/4/04 

t 

First  day 

t 

First  day 

Last  day  of  school 

t 

5/24/05 

t 

5/23/062 

6/7/072'3 

Data  collection 
Start 

8/30/04 

4/11/05 

9/26/054 

4/24/06 

4/26/07 

End 

9/1 6/04 

4/29/05 

1 0/20/05 

5/5/06 

5/1 0/07 

Calendar  weeks  from 
program  implementation 
to  start  of  fall  2004  data 
collection 

4 

t 

t 

t 

t 

Calendar  weeks  from 
start  of  school  to  start 
of  fall  data  collection 

51 

t 

5 

t 

t 

Calendar  weeks  from  end 
of  fall  data  collection  to 
start  of  spring  data 
collection 

t 

30 

t 

27 

t 

Calendar  weeks  from 
program  implementation 
to  start  of  spring  data 
collection 

t 

36 

t 

35 

352 

t Not  applicable. 

1 Five  schools  started  on  7/26/04  and  five  started  on  8/2/04.  The  calculation  of  weeks  from  the  start  of  school  to  the  start  of  fall  2004 
data  collection  uses  7/26/04  as  the  first  day  of  the  school  year. 

2 During  the  2005-06  school  year  and  the  2006-07  school  year,  one  school  operated  on  a year-round  schedule. 

3 Schools  ended  between  6/7/07  and  6/8/07,  with  most  ending  on  6/7/07. 

4 Data  collection  at  one  school  occurred  3 weeks  earlier. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Consent  Rates,  Completion  Rates,  and  Percentage  of  Sample  With  Data 

The  actual  number  of  student,  primary  caregiver,  and  teacher  reports  available  for  analysis  was  smaller  than 
the  number  in  the  sample  because  consent  and  completion  rates  were  less  than  100  percent.  Primary 
caregivers  had  to  provide  consent  before  children  could  complete  the  Child  Report,  before  their  child’s 
teacher  could  complete  the  Teacher  Report  on  Student,  and  before  they  themselves  completed  the  Primary 
Caregiver  Report.  Teachers  also  had  to  provide  consent  before  completing  the  Teacher  Report  on  Classroom 
and  School.  Of  those  with  consent,  not  all  completed  their  respective  reports.  Table  3.4  shows  the  consent 
rates,  completion  rates,  and  percentages  of  sample  with  data  for  each  of  the  four  reports  over  the  3 years.  For 
the  Child  Report  and  the  Teacher  Report  on  Student,  completion  rates  ranged  from  91  percent  to  100 
percent.  In  Year  3,  a significandy  higher  percentage  of  control  than  treatment  students  had  consent  for  all 
three  reports  on  students.  For  the  Primary  Caregiver  Report,  the  completion  rates  dropped  over  time  from  91 
percent  to  74  percent. 

The  percentages  of  the  sample  with  Child  Report  data  ranged  from  59  percent  to  71  percent  over  the  3 years, 
with  a significantly  higher  percentage  of  control  students  having  data  in  Year  3.  The  percentages  of  students 
with  information  from  the  Teacher  Report  on  Student  similarly  ranged  from  55  percent  to  70  percent,  with  a 
significantly  higher  percentage  of  control  students  having  data  in  Year  3.  The  percentages  of  students  with 
data  from  the  Primary  Caregiver  Report  ranged  from  43  percent  to  63  percent  and  declined  over  time.  In 
Year  1,  the  percentage  of  primary  caregivers  with  data  was  significandy  less  in  the  treatment  group  than  in  the 
control  group  (56%  versus  63%).  The  percentages  of  teachers  with  data  from  the  Teacher  Report  on 
Classroom  and  School  ranged  from  78  percent  to  95  percent. 
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Table  3.4.  Consent  rates,  completion  rates,  and  percentage  of  sample  with  data  from  each  report — CSP 


Year  1 

Year  1 

Year  2 

Year  3 

(Fall  3rd  grade) 

(Spring  3rd  grade) 

(Spring  4th  grade) 

(Spring  5th  grade) 

Report 

T reat- 

Total  ment  Control 

T reat- 

Total  ment  Control 

T reat- 

Total  ment  Control 

T reat- 

Total  ment  Control 

Student  sample  size 

959 

476 

483 

975 

485 

490 

969 

474 

495 

947 

4583 

489 

Child  Report  (percent) 

Primary  caregiver  consent  rate 

70.0 

68.7 

71.2 

69.4 

68.7 

70.2 

68.5 

67.3 

69.7 

66.4 

60.0*** 

72.4 

Student  completion  rate 

92.5 

91.4 

93.6 

97.6 

97.9 

97.4 

97.0 

98.4 

95.7 

98.4 

98.5 

98.3 

Students  with  data1 

64.8 

62.8 

66.6 

67.8 

67.2 

68.4 

66.5 

66.2 

66.7 

65.4 

59.2*** 

71.2 

Primary  Caregiver  Report  (percent) 

Primary  caregiver  consent  rate 

65.9 

63.4 

68.3 

67.4 

66.6 

68.2 

65.1 

63.9 

66.3 

62.8 

57.0*** 

68.3 

Primary  caregiver  completion  rate 

90.5 

88.7 

92.1 

80.1 

76.5* 

83.5 

80.0 

78.5 

81.4 

74.3 

76.2 

72.8 

Primary  caregivers  with  data1 

59.6 

56.3* 

62.9 

53.9 

50.9 

56.9 

52.1 

50.2 

53.9 

46.7 

43.4 

49.7 

Teacher  Report  on  Student  (percent) 

Primary  caregiver  consent  rate2 

70.0 

68.7 

71.2 

69.4 

68.7 

70.2 

68.5 

67.3 

69.7 

66.4 

60.0*** 

72.4 

Teacher  completion  rate 

94.9 

93.3* 

96.5 

99.0 

98.8 

99.1 

99.7 

100.0 

99.4 

94.6 

91.6*** 

96.9 

Students  with  data1 

66.4 

64.1 

68.7 

68.7 

67.8 

69.6 

68.3 

67.3 

69.3 

62.8 

55.0*** 

70.1 

Teacher  Report  on  Classroom  and  School 
(3rd-  to  5th-  grade  teachers)  (percent) 

Teacher  consent  rate 

100.0 

100.0 

100.0 

100.0 

100.0 

100.0 

99.2 

98.5 

100.0 

100.0 

100.0 

100.0 

Teacher  completion  rate 

84.1 

83.1 

85.2 

78.5 

77.9 

79.0 

88.6 

86.4 

90.9 

90.6 

86.3 

95.4 

Teachers  with  data1 

84.1 

83.1 

85.2 

78.5 

77.9 

79.0 

88.0 

85.1 

90.9 

90.6 

86.3 

95.4 

* Treatment  group  significantly  different  from  control  group  at  the  .05  level. 

***  Treatment  group  significantly  different  from  control  group  at  the  .001  level. 

1 Calculated  as  consent  rate  x completion  rate. 

2The  primary  caregiver  consent  rates  for  the  Child  Report  and  the  Teacher  Report  on  Student  are  identical,  as  the  primary  caregiver  gave  consent  to  both  together. 

3 One  of  the  treatment  group  schools  that  had  been  a K-5  school  in  prior  years  became  a K-4  school  and  continued  implementation  of  the  intervention.  The  Cohort  1 students,  who 
were  in  fifth  grade  during  the  2006-07  school  year,  went  to  an  intermediate  school  that  had  fifth  and  sixth  grades.  The  intermediate  school  did  not  implement  the  intervention.  These 
fifth-grade  students,  their  primary  caregivers,  and  their  teachers  were  included  in  the  spring  2007  data  collection  and  all  analyses  based  upon  spring  2007  data. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 


Chapter  3.  Competence  Support  Program 


Chapter  3.  Competence  Support  Program 


Responses  from  students  in  the  original  cohort  (stayers)  and  new  entrants  in  the  CSP  sample  were  examined 
to  investigate  possible  differences  between  the  two  groups  in  consent  rates,  completion  rates,  and  the 
percentages  of  sample  with  data  that  might  affect  outcome  data  (table  3.5).  In  Year  1,  stayers  completed  more 
Child  Reports  than  did  new  entrants  (98%  versus  93%),  a statistically  significant  difference.  In  Years  2 and  3, 
primary  caregivers  of  the  stayers  had  significantly  higher  consent  rates  for  both  child  and  primary  caregiver, 
which  led  to  significantly  higher  percentages  of  data  on  all  three  reports  in  these  2 years  for  the  stayers.  The 
difference  in  consent  rates  for  these  two  groups  was  significandy  different  by  14  to  35  percentage  points. 
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Table  3.5.  Consent  rates,  completion  rates,  and  percentage  of  sample  with  data:  Stayers  versus  new  entrants — CSP 


Year  1 Year  2 Year  3 

(Spring  3rd  grade) (Spring  4th  grade) (Spring  5th  grade) 

New  New  New 


Report 

Total 

Stayers 

entrants 

Total 

Stayers 

entrants 

Total 

Stayers 

entrants 

Student  sample  size 

975 

923 

52 

969 

729 

240 

947 

721 

226 

Child  Report  (percent) 

Primary  caregiver  consent  rate 

69.4 

69.0 

76.9 

68.5 

2*** 

42.1 

66.4 

69.8*** 

55.8 

Student  completion  rate 

97.6 

98.0* 

92.5 

97.0 

97.0 

97.0 

98.4 

98.4 

98.4 

Students  with  data1 

67.8 

67.6 

71.2 

66.5 

74.9*** 

40.8 

65.4 

68.7*** 

54.9 

Primary  Caregiver  Report  (percent) 

Primary  caregiver  consent  rate 

67.4 

67.0 

75.0 

65.1 

73.8*** 

38.8 

62.8 

66.2*** 

52.2 

Primary  caregiver  completion  rate 

80.1 

80.1 

79.5 

80.0 

81.2 

73.1 

75.1 

75.1 

75.4 

Primary  caregivers  with  data1 

53.9 

53.6 

59.6 

52.1 

59.9*** 

28.3 

47.2 

49.7** 

39.4 

Teacher  Report  on  Student  (percent) 

Primary  caregiver  consent  rate2 

69.4 

69.0 

76.9 

68.5 

2*** 

42.1 

66.4 

69.8*** 

55.8 

Teacher  completion  rate 

99.0 

98.9 

100.0 

99.7 

99.8 

99.0 

94.6 

94.6 

94.4 

Students  with  data1 

68.7 

68.3 

76.9 

68.3 

77.1*** 

41.7 

62.8 

66.0*** 

52.7 

* Stayers  significantly  different  from  new  entrants  at  the  .05  level. 

**  Stayers  significantly  different  from  new  entrants  at  the  .01  level. 

***  Stayers  significantly  different  from  new  entrants  at  the  .001  level. 

1 Calculated  as  consent  rate  x completion  rate. 

2The  primary  caregiver  consent  rates  for  the  Child  Report  and  the  Teacher  Report  on  Student  are  identical,  as  the  primary  caregiver  gave  consent  to  both  together. 
SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Fidelity  of  Implementation 

Each  year,  CSP’s  five  treatment  schools  were  independently  rated  for  quantity  and  quality  of  the  program’s 
implementation  by  two  raters  from  the  research  team.  The  global  measure  of  fidelity  for  the  multisite  study 
was  used;  inter-rater  reliability  was  measured  using  Cronbach’s  alpha  (0.70,  0.80,  and  0.99  across  the  3 years). 
The  ratings  were  combined  into  a single  consensus  rating  and  used  to  identify  schools  with  high 
implementation  fidelity.  In  all  3 years,  two  treatment  schools  were  identified  as  having  high  fidelity.  Cohen’s 
kappa  was  used  as  the  measure  of  agreement  when  identifying  schools  as  high  fidelity,  and  it  equaled  0.62  in 
Year  1,  0.55  in  Year  2,  and  1.00  in  Year  3. 

Initial  Characteristics 

This  section  examines  the  initial  characteristics  of  the  students,  teachers,  and  schools  participating  in  the  CSP 
evaluation.  These  characteristics  were  collected  from  students  who  were  enrolled  in  the  third  grade  at  the 
study  schools  in  fall  2004,  as  well  as  from  their  primary  caregivers  and  third-grade  teachers.  In  addition, 
third-,  fourth-,  and  fifth-grade  teachers  and  principals  in  the  study  schools  provided  information  about 
activities  related  to  social  and  character  development  in  these  schools.  Documenting  the  characteristics  of 
students,  teachers,  and  schools  and  initial  measures  of  key  outcomes  at  a point  before  the  intervention  had 
been  operating  for  an  extended  period  helps  to  determine  whether  the  random  assignment  of  schools  to 
treatment  and  control  status  produced  treatment  and  control  groups  with  similar  distributions  of  observed 
characteristics.  As  noted  in  the  following  discussion,  7 significant  differences  were  found  in  the  observed 
characteristics,  6 of  which  (out  of  62  comparisons,  with  3 expected  to  be  significant  by  chance)  related  to 
differences  between  the  treatment  and  control  groups  in  the  level  of  SACD  activity  in  the  classroom  and 
school. 

Characteristics  of  Children,  Their  Families,  and  Communities 

There  were  no  significant  differences  in  the  observed  student,  caregiver,  and  community  characteristics 
between  the  treatment  and  control  groups  (see  table  3.6).  For  students,  the  mean  age  was  8.1  years.  The 
sample  contained  roughly  equal  percentages  of  girls  (53%)  and  boys  (47%).  The  sample  was  ethnically 
diverse.  White  non-Hispanic  students  made  up  47  percent  of  the  sample,  Black  non-Hispanic  students  made 
up  35  percent,  and  Hispanic  students  made  up  10  percent. 

The  sample  was  also  diverse  in  its  levels  of  family  income,  the  education  levels  of  primary  caregivers  of  the 
children  in  the  sample,  and  family  situation.  Thirty-three  percent  of  the  children  lived  in  a household  where 
the  income  was  1 35  percent  of  the  federal  poverty  level  or  lower,  which  is  the  income  threshold  for  eligibility 
for  free  school  meals.  Ten  percent  of  the  primary  caregivers  had  not  completed  high  school.  About  two- 
thirds  of  the  children  (65%)  lived  with  both  their  mother  and  their  father.  There  were  no  significant 
differences  between  the  treatment  and  control  groups  in  these  characteristics. 

The  mean  values  of  the  outcomes  for  children’s  behavior  and  attitudes  as  reported  by  the  primary  caregiver, 
child,  and  teacher  in  fall  2004  are  shown  in  table  3.7.  There  was  1 significant  difference  (out  of  18 
comparisons)  in  these  scores  between  the  treatment  and  control  groups;  children  in  the  treatment  group 
reported  a more  negative  school  orientation  than  did  those  in  the  control  group. 
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Table  3.6.  Initial  characteristics  of  children,  their  families,  and  communities — CSP 


Characteristic 

Total 

Treatment 

Control 

Student  sample  size 

572 

268 

304 

Student  demographics 

Gender  (percent) 

Male 

46.9 

45.2 

48.7 

Female 

53.1 

54.8 

51.3 

Race/ethnicity  (percent) 

White  (non-Hispanic) 

46.6 

35.4 

57.7 

Black  (non-Hispanic) 

34.5 

43.5 

25.4 

Hispanic 

9.9 

8.7 

11.1 

Other 

9.1 

12.4 

5.7 

Age  (in  years)  (mean) 

8.1 

8.1 

8.1 

Primary  caregiver  and  family  characteristics 

Primary  caregiver’s  age  (in  years)  (mean) 

34.7 

34.3 

35.1 

Primary  caregiver’s  race/ethnicity  (percent) 

White  (non-Hispanic) 

51.2 

41.0 

61.3 

Black  (non-Hispanic) 

33.4 

42.7 

24.1 

Hispanic 

8.6 

6.8 

10.4 

Other 

6.9 

9.5 

4.2 

Primary  caregiver’s  education  (percent) 

Did  not  complete  high  school 

9.5 

8.0 

11.0 

Completed  high  school  or  equivalent 

27.7 

25.4 

30.1 

Some  college 

47.3 

51.0 

43.5 

Bachelor’s  or  higher  degree 

15.5 

15.6 

15.4 

Primary  caregiver’s  employment  (percent) 

Full-time 

54.1 

51.5 

56.7 

Other 

45.9 

48.5 

43.3 

Primary  caregiver’s  marital  status  (percent) 

Married 

64.2 

60.7 

67.7 

Other 

35.8 

39.3 

32.3 

Students  who  live  in  one  household  (percent) 

95.2 

95.9 

94.6 

Number  of  individuals  in  household  (mean) 

4.5 

4.4 

4.5 

Primary  caregiver’s  relationship  to  child  (percent) 

Mother  (stepmother) 

85.7 

84.8 

86.5 

Father  (stepfather) 

8.0 

7.8 

8.2 

Other  relative/nonrelative 

6.3 

7.4 

5.2 

See  notes  at  end  of  table. 
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Table  3.6.  Initial  characteristics  of  children,  their  families,  and  communities — CSP — Continued 


Characteristic 

Total 

Treatment 

Control 

Student  lives  with  (percent) 

Mother  (stepmother)  and  father  (stepfather) 

64.6 

60.3 

68.8 

Mother  (stepmother)  only;  father  (stepfather)  not  present 

28.5 

32.3 

24.8 

Father  (stepfather)  only;  mother  (stepmother)  not  present 

1.6 

1.4 

1.8 

Other  relative/nonrelative,  parents  not  present 

5.3 

6.0 

4.6 

Highest  education  of  anyone  in  household  (percent) 

Did  not  complete  high  school 

6.8 

7.3 

6.3 

Completed  high  school  or  equivalent 

21.1 

17.5 

24.7 

Some  college 

51.5 

54.6 

48.4 

Bachelor’s  or  higher  degree 

20.7 

20.7 

20.7 

Total  household  income  (percent) 

Less  than  $20,000 

29.1 

30.3 

27.9 

$20,000  to  $39,999 

31.7 

35.3 

28.2 

$40,000  to  $59,999 

21.2 

18.8 

23.6 

$60,000  or  more 

18.0 

15.6 

20.3 

Income-to-poverty-threshold  ratio — Below  135  percent  (percent) 

33.1 

34.1 

32.0 

Income-to-poverty-threshold  ratio — 135  to  185  percent  (percent) 

25.1 

26.3 

24.0 

Income-to-poverty-threshold  ratio — Above  185  percent  (percent) 

41.8 

39.6 

44.0 

Alabama  Parenting  Questionnaire — Poor  Monitoring  and 

Supervision  Subscale  (mean) 

1.2 

1.2 

1.2 

Alabama  Parenting  Questionnaire — Positive  Parenting 

Subscale  (mean) 

3.6 

3.6 

3.6 

Confusion,  Hubbub,  and  Order  Scale  (mean) 

2.2 

2.1 

2.2 

Community  characteristics  (mean) 

Community  Risks  Scale 

1.2 

1.2 

1.2 

Community  Resources  Scale 

2.2 

2.3 

2.2 

Child-Centered  Social  Control  Scale 

3.2 

3.1 

3.2 

NOTE:  No  statistically  significant  differences  were  found  between  values  for  treatment  and  control  groups.  Weights,  which  assign 
equal  weight  to  each  school  within  the  program,  were  used  in  producing  the  treatment,  control,  and  overall  means.  Statistical  tests 
were  conducted  using  regressions  that  included  program  indicators  to  account  for  the  sample  design  and  adjusted  for  clustering  at 
the  school  level. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Table  3.7.  Mean  scores  and  standard  deviations  for  initial  outcome  measures  of  sample — CSP 


Outcome  measure-Report 

Range 

Total 

Mean 

SD 

Treatment 
Mean  SD 

Control 
Mean  SD 

Social  and  Emotional  Competence  Domain 

Self-Efficacy  for  Peer  Interaction-CR 

1-4 

2.9 

0.6 

2.9 

0.6 

3.0 

0.6 

Normative  Beliefs  About  Aggression-CR 

1-4 

1.2 

0.4 

1.2 

0.4 

1.2 

0.4 

Empathy-CR 

1-3 

2.4 

0.4 

2.4 

0.4 

2.4 

0.4 

Behavior  Domain 

Altruistic  Behavior-CR 

0-3 

1.5 

0.8 

1.5 

0.9 

1.4 

0.8 

Altruistic  Behavior-TRS 

1-4 

1.4 

0.5 

1.5 

0.5 

1.3 

0.4 

Altruistic  Behavior-PCR 

1-4 

2.3 

0.7 

2.3 

0.7 

2.3 

0.7 

Positive  Social  Behavior-TRS 

1-4 

3.0 

0.6 

3.0 

0.6 

3.0 

0.6 

Positive  Social  Behavior-PCR 

1-4 

3.0 

0.5 

3.0 

0.5 

3.0 

0.5 

Problem  Behavior-CR 

0-3 

0.2 

0.4 

0.2 

0.4 

0.2 

0.4 

Problem  Behavior-TRS 

1-4 

1.4 

0.4 

1.4 

0.4 

1.3 

0.3 

Problem  Behavior-PCR 

1-4 

1.5 

0.3 

1.5 

0.3 

1.6 

0.3 

ADHD-Related  Behavior-TRS 

1-4 

1.8 

0.7 

1.8 

0.7 

1.7 

0.6 

Academics  Domain 

Academic  Competence  and  Motivation-TRS 

1-5 

3.0 

0.8 

3.0 

0.8 

2.9 

0.9 

Engagement  with  Learning-CR 

1-4 

3.7 

0.7 

3.6 

0.7 

3.7 

0.6 

Perceptions  of  School  Climate  Domain 

Positive  School  Orientation-CR 

1-4 

3.1 

0.7 

3.1 

0.7 

3.1 

0.7 

Negative  School  Orientation-CR 

1-4 

1.9 

0.6 

2.0* 

' 0.7 

1.8 

0.6 

Student  Afraid  at  School-CR 

1-4 

2.5 

0.9 

2.5 

0.9 

2.5 

0.9 

Victimization  at  School-CR 

0-3 

0.8 

0.8 

0.9 

0.8 

0.8 

0.8 

Student  sample  size — PCR 

572 

268 

304 

Student  sample  size — CR 

621 

299 

322 

Student  sample  size — TRS 

638 

306 

332 

* Treatment  group  significantly  different  from  control  group  at  the  .05  level. 
NOTE:  Abbreviations  are 


CR:  Child  Report 

PCR:  Primary  Caregiver  Report 

TRS:  Teacher  Report  on  Student 

ADHD:  Attention  deficit  hyperactivity  disorder 

SD:  Standard  deviation 

Weights,  which  assign  equal  weight  to  each  school  within  the  program,  were  used  in  producing  the  treatment,  control,  and  overall 
means.  Statistical  tests  were  conducted  using  regressions  that  included  program  indicators  to  account  for  the  sample  design  and 
adjusted  for  clustering  at  the  school  level.  Sample  size  may  differ  for  some  outcomes  due  to  nonresponse. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Characteristics  of  Teachers  and  Schools 

The  third-,  fourth-,  and  fifth-grade  teachers  at  the  study  schools  were  predominandy  White  non-Hispanic 
(77%)  and  female  (93%),  had  an  average  of  14.8  years  of  total  teaching  experience,  and  about  26  percent  held 
an  advanced  or  specialist  degree  (see  table  3.8).  There  was  one  statistically  significant  difference  between  the 
groups  of  teachers;  control  group  teachers  had  been  teaching  in  their  current  schools  for  a greater  number  of 
years  on  average  than  had  treatment  group  teachers  (7.7  versus  4.9  years). 

Data  regarding  school  characteristics  were  drawn  from  the  Common  Core  of  Data  in  order  to  compare 
treatment  and  control  schools.  There  were  no  significant  differences  between  the  two  groups  of  schools  in 
terms  of  student  composition  (race/ ethnicity  and  school  lunch  eligibility),  number  of  students  enrolled, 
number  of  full-time  teachers,  Title  I status,  or  number  of  years  the  principal  had  been  at  the  school  (table 
3.9).  In  addition,  there  were  no  significant  differences  between  treatment  and  control  schools  in  terms  of 
location  (urban,  suburban,  or  rural)  or  lowest  and  highest  grade  offered  (these  data  are  not  shown  in  a table). 


Table  3.8.  Initial  characteristics  of  teachers  in  sample — CSP 


Characteristic 

Total 

Treatment 

Control 

Teacher  sample  size 

106 

54 

52 

Gender  (percent) 

Male 

7.2 

7.1 

7.4 

Female 

92.8 

92.9 

92.6 

Race/ethnicity  (percent) 

White  (non-Hispanic) 

77.1 

74.8 

79.4 

Other 

22.9 

25.2 

20.6 

Number  of  years  teaching  experience  (mean) 

14.8 

14.7 

14.9 

Number  of  years  teaching  experience  in  this  school  (mean) 

6.3 

4.9* 

7.7 

Type  of  teaching  certificate  (percent) 

Regular  state  certificate  or  advanced  professional  certificate 

95.0 

t 

t 

Other 

5.0 

t 

t 

Education  (percent) 

Bachelor’s  degree 

74.3 

66.7 

81.9 

Advanced  degree/other 

25.7 

33.3 

18.1 

$ Reporting  standards  not  met.  Values  suppressed  to  protect  confidentiality. 

* Treatment  group  significantly  different  from  control  group  at  the  .05  level. 

NOTE:  Weights,  which  assign  equal  weight  to  each  school  within  the  program,  were  used  in  producing  the  treatment,  control,  and 
overall  means.  Statistical  tests  were  conducted  using  regressions  that  included  program  indicators  to  account  for  the  sample  design 
and  adjusted  for  clustering  at  the  school  level.  Sample  size  may  differ  for  some  outcomes  due  to  nonresponse. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Table  3.9.  Initial  characteristics  of  schools  in  sample — CSP 


Characteristic 

Total 

Treatment 

Control 

School  sample  size 

10 

5 

5 

Student  race/ethnicity  (percent) 

White  (non-Hispanic) 

44.0 

34.8 

53.3 

Black  (non-Hispanic) 

41.3 

51.0 

31.5 

Hispanic 

10.3 

8.0 

12.7 

Other 

4.4 

6.2 

2.5 

Students  eligible  for  free  or  reduced-price  lunch  (percent) 

57.0 

63.8 

50.2 

Number  of  students  enrolled  (mean) 

626.5 

657.8 

595.2 

Number  of  full-time  teachers  (mean) 

38.9 

40.8 

37.0 

Title  1 status  (percent) 

Title  1 eligible  school 

100.0 

100.0 

100.0 

Schoolwide  Title  1 

100.0 

100.0 

100.0 

Number  of  years  principal  has  been  at  this  school  (mean) 

4.9 

2.5 

6.8 

NOTE:  No  statistically  significant  differences  were  found  between  values  for  treatment  and  control  groups.  Weights,  which  assign 
equal  weight  to  each  school  within  the  program,  were  used  in  producing  the  treatment,  control,  and  overall  means.  Statistical  tests 
were  conducted  using  regressions  that  included  program  indicators  to  account  for  the  sample  design  and  adjusted  for  clustering  at 
the  school  level.  Sample  size  may  differ  for  some  outcomes  due  to  nonresponse. 

SOURCE:  NCES  Common  Core  of  Data  (2003-04),  the  Social  and  Character  Development  (SACD)  Research  Program. 


In  the  Teacher  Report  on  Classroom  and  School,  teachers  reported  on  nine  dimensions  of  school 
environment  (these  data  are  not  shown  in  a table):  feelings  of  safety,  adequacy  of  resources,  student  support, 
freedom  to  teach  as  desired,  affiliation  with  and  ties  to  colleagues,  innovation  regarding  new  approaches  to 
teaching,  professional  interest,  participatory  decisionmaking,  and  work  pressure.  There  were  no  statistically 
significant  differences  between  treatment  and  control  schools  in  these  reports. 

The  Level  of  SACD  in  the  Schools  Near  the  Beginning  of  the  Study 

During  the  initial  data  collection,  principals  and  teachers  reported  on  the  SACD  activities  used  in  the  schools 
and  classrooms,  the  availability  of  SACD  materials,  and  the  professional  development  provided  on  SACD.  It 
is  important  to  note  that  this  data  collection  at  the  beginning  of  the  school  year  took  place,  on  average,  4 
weeks  after  the  start  of  the  CSP  implementation;  therefore  the  activities  and  practices  related  to  SACD  in  the 
treatment  and  control  schools  are  a measure  of  what  was  happening  in  the  study  schools  near  the  beginning 
of  the  study.  Table  3.10  shows  that  the  majority  of  the  school  principals  reported  activities  to  promote  six 
social  and  character  development  goals:  violence  prevention  and  peace  promotion  (100%),  social  and 
emotional  development  (100%),  character  education  (100%),  tolerance  and  diversity  (80%),  risk  prevention 
and  health  promotion  (80%),  and  civic  responsibility  and  community  service  (90%).  In  addition,  all  of  the 
principals  reported  activities  directed  toward  behavior  management.  There  were  no  statistically  significant 
differences  between  the  treatment  group  and  the  control  group  in  the  percentages  on  principal  reports, 
although  this  may  be  due  to  the  relatively  small  principal  sample  size.  Teachers’  reports  on  the  use  of  these 
activities  in  their  classrooms  ranged  from  60  percent  to  93  percent,  with  no  significant  difference  between 
treatment  and  control  teachers.  With  respect  to  the  use  of  schoolwide  activities,  64  percent  to  97  percent  of 
teachers  reported  that  their  schools  used  such  activities,  and  again  there  were  no  significant  differences 
between  treatment  and  control  teachers  in  reported  use. 
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Table  3.10.  Principal  and  teacher  initial  reports  on  use  of  SACD  programs  or  activities  in 
sample — CSP 


SACD  program  or  activity 

Total 

Treatment 

Control 

Principal  sample  size 

10 

5 

5 

Teacher  sample  size 

106 

54 

52 

Principals  reporting  that  school  had  programs  or  activities 
to  promote  the  following  SACD  goals  (percent) 

Violence  prevention  and  peace  promotion 

100.0 

100.0 

100.0 

Social  and  emotional  development 

100.0 

100.0 

100.0 

Character  education 

100.0 

100.0 

100.0 

Tolerance  and  diversity 

80.0 

80.0 

80.0 

Risk  prevention  and  health  promotion 

80.0 

100.0 

60.0 

Civic  responsibility  and  community  service 

90.0 

80.0 

100.0 

Behavior  management 

100.0 

100.0 

100.0 

None  of  the  above 

0.0 

0.0 

0.0 

Teachers  reporting  on  using  programs  or  activities  in  their 
class  to  promote  the  following  SACD  goals  (percent) 

Violence  prevention  and  peace  promotion 

60.2 

67.9 

52.4 

Social  and  emotional  development 

76.5 

77.4 

75.6 

Character  education 

84.6 

88.2 

80.9 

Tolerance  and  diversity 

66.4 

67.3 

65.6 

Risk  prevention  and  health  promotion 

67.4 

60.8 

74.1 

Civic  responsibility  and  community  service 

62.1 

62.7 

61.6 

Behavior  management 

93.4 

94.1 

92.7 

None  of  the  above 

0.0 

0.0 

0.0 

Teachers  reporting  schoolwide  use  of  the  following 
activities  to  promote  SACD  (percent) 

Morning  announcements  or  videos 

97.2 

95.6 

98.8 

School  assemblies 

75.8 

74.2 

77.3 

School  newspapers  or  bulletins 

73.5 

67.4 

79.6 

Special  school  days 

63.9 

57.7 

70.0 

Special  events 

80.5 

83.0 

78.1 

Other  activities 

17.2 

18.9 

15.6 

NOTE:  No  statistically  significant  differences  were  found  between  values  for  treatment  and  control  groups.  Weights,  which  assign 
equal  weight  to  each  school  within  each  of  the  programs  and  to  each  program  across  programs,  were  used  in  producing  the 
treatment,  control,  and  overall  means.  Statistical  tests  were  conducted  using  regressions  that  included  program  indicators  to 
account  for  the  sample  design  and  adjusted  for  clustering  at  the  school  level.  Sample  size  may  differ  for  some  outcomes  due  to 
nonresponse. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 

Teachers  reported  using  a broad  range  of  teaching  materials  to  support  SACD  activities  (table  3.11),  including 
teacher  guides  (69%),  student  materials  (69%),  giveaways  (50%),  instructional  aids  (45%),  and  children’s 
literature  (42%).  There  were  statistically  significant  differences  in  the  use  of  teacher  guides  (81%  versus  57%) 
and  instructional  aids  (53%  versus  38%),  with  treatment  teachers  reporting  greater  use  of  these  two  types  of 
materials  than  control  teachers. 

Teachers  also  reported  using  a wide  variety  of  teaching  strategies  (table  3.11).  All  teachers  reported  using  any 
of  the  20  strategies  asked  about,  and  teachers  used  an  average  of  12.3  of  the  strategies.  There  were  no 
significant  differences  in  the  average  number  of  strategies,  or  in  the  specific  strategies,  used  by  treatment 
versus  control  teachers. 
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Table  3.11.  Teacher  initial  reports  on  use  of  SACD  materials  and  classroom  strategies  in 
sample — CSP 


SACD  material  and  classroom  strategy 

Total 

Treatment 

Control 

Teacher  sample  size 

106 

54 

52 

Teachers  using  the  following  materials  in  conjunction  with 
social  and  character  development  activities  (percent) 

Teacher  guides  (manuals,  curricula) 

69.0 

81.0* 

57.0 

Student  materials  (workbooks,  worksheets) 

68.8 

74.4 

63.1 

Instructional  aids  (games,  software,  videos) 

45.3 

53.0* 

37.6 

Giveaways  (bookmarks,  stickers) 

50.3 

49.9 

50.8 

Children’s  literature 

42.3 

35.5 

49.0 

Other  types  of  materials 

14.6 

12.8 

16.3 

Do  not  use  any  of  the  materials  listed  above 

8.3 

t 

t 

Teachers  using  any  of  the  strategies  listed  below  to  promote 

social  and  character  development  in  the  classroom  (percent) 

100.0 

100.0 

100.0 

Number  of  strategies  (listed  below)  used  by  teachers  to  promote 

social  and  character  development  in  the  classroom  (mean) 

12.3 

13.1 

11.6 

Teachers  using  each  of  the  following  strategies  to  promote 
social  and  character  development  (percent) 

Role-playing 

62.4 

68.1 

56.8 

Cooperative  learning 

92.9 

93.0 

92.7 

Peer  group  discussions 

86.7 

86.8 

86.7 

Direct  instruction  of  social  and  character  development 

85.7 

87.9 

83.5 

Skill  training 

30.2 

40.1 

20.2 

Incorporating  social  and  character  development  into 

academic  curriculum 

82.2 

89.6 

74.8 

Parent  training 

18.7 

12.2 

25.1 

Parent/community  involvement  in  program  development 

or  delivery 

30.5 

35.7 

25.3 

Mentoring 

38.7 

44.5 

32.9 

Good  behavior  notes  sent  home  daily  or  weekly 

90.3 

85.5 

95.1 

Presenting  role  models 

71.9 

64.2 

79.5 

Targeted  story  reading  or  writing  on  SACD  themes 

75.0 

82.5 

67.5 

Peer  mediation 

48.8 

56.8 

40.8 

Honor  roll  for  positive  behavior 

57.2 

54.0 

60.3 

Pledges  or  recitations  on  social  and  character 

development  themes 

42.2 

53.3 

31.1 

Guided  visualization 

54.4 

61.6 

47.2 

Student-led/student-assisted  instruction 

45.0 

54.9 

35.0 

Journaling 

58.9 

57.4 

60.5 

Time  out  for  negative  behavior 

94.4 

98.3 

90.5 

Daily  or  weekly  rewards  for  positive  behavior 

96.5 

95.6 

97.5 

4 Values  suppressed  to  protect  confidentiality. 

* Treatment  group  significantly  different  from  control  group  at  the  .05  level. 

NOTE:  Weights,  which  assign  equal  weight  to  each  school  within  each  of  the  programs  and  to  each  program  across  programs,  were 
used  in  producing  the  treatment,  control,  and  overall  means.  Statistical  tests  were  conducted  using  regressions  that  included 
program  indicators  to  account  for  the  sample  design  and  adjusted  for  clustering  at  the  school  level.  Sample  size  may  differ  for  some 
outcomes  due  to  nonresponse. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Principals  and  teachers  reported  on  participation  in  and  amount  of  SACD  training  and  staff  development 
provided  over  the  previous  12  months  (table  3.12).  Principals  reported  higher  participation  rates  (100% 
versus  72%)  and  reported  more  training  hours  (8.9  versus  7.2)  than  did  teachers.  There  was  a significant 
difference  in  the  percentages  of  teachers  reporting  participation  in  training,  with  more  treatment  teachers 
reporting  participation  in  SACD  training  than  control  teachers  (93%  versus  52%).  In  addition,  a significandy 
greater  percentage  of  treatment  teachers  than  control  teachers  reported  receiving  training  in  three  of  the 
seven  targeted  SACD  goals:  social  and  emotional  development  (38%  versus  9%),  character  education  (62% 
versus  14%),  and  behavior  management  (60%  versus  26%). 


Table  3.12.  Principal  and  teacher  initial  reports  on  SACD  professional  development  in  sample — 
CSP 


SACD  professional  development 

Total 

Treatment 

Control 

Principal  sample  size 

10 

5 

5 

Teacher  sample  size 

106 

54 

52 

Principals  reporting  that  staff  participated  in  social  and  character 
development  training  within  the  past  year  (percent) 

100.0 

100.0 

100.0 

Teachers  reporting  participation  in  social  and  character  development 
training  within  the  past  12  months  (percent) 

72.2 

93.0** 

51.5 

Number  of  hours  of  social  and  character  development  training 
principals  report  were  provided  to  each  staff  person  last  year  (mean) 

8.9 

9.6 

8.2 

Number  of  hours  of  social  and  character  development  training 
teachers  report  receiving  during  the  past  12  months  (mean) 

7.2 

8.1 

6.4 

Teachers  reporting  receiving  training  in  the  past  12  months 
in  the  following  areas  (percent) 

Violence  prevention  and  peace  promotion 

7.9 

t 

t 

Social  and  emotional  development 

23.3 

37.8* 

8.8 

Character  education 

37.6 

61.6** 

13.6 

Tolerance  and  diversity 

20.5 

24.0 

17.0 

Risk  prevention  and  health  promotion 

8.6 

11.9 

5.3 

Civic  responsibility  and  community  service 

3.0 

t 

i 

Behavior  management 

43.1 

60.0* 

26.1 

t Reporting  standards  not  met.  Values  suppressed  to  protect  confidentiality. 


* Treatment  group  significantly  different  from  control  group  at  the  .05  level. 

**  Treatment  group  significantly  different  from  control  group  at  the  .01  level. 

NOTE:  Weights,  which  assign  equal  weight  to  each  school  within  each  of  the  programs  and  to  each  program  across  programs,  were 
used  in  producing  the  treatment,  control,  and  overall  means.  Statistical  tests  were  conducted  using  regressions  that  included 
program  indicators  to  account  for  the  sample  design  and  adjusted  for  clustering  at  the  school  level.  Sample  size  may  differ  for  some 
outcomes  due  to  nonresponse. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 


The  data  on  the  initial  levels  of  SACD  activity  emphasize  that  the  control  condition  was  a “standard  practice” 
control.  Standard  practice  at  the  control  schools  included  using  SACD  activities,  materials,  and  practices, 
along  with  professional  development  for  staff,  at  rates  and  in  types  and  amounts  similar  to  the  treatment 
schools.  For  example,  the  percentages  of  teachers  who  reported  using  programs  or  activities  to  promote 
specific  SACD  goals  ranged  from  61  percent  to  94  percent  in  the  treatment  schools  and  from  52  percent  to 
93  percent  in  the  control  schools.  However,  the  6 significant  differences  between  the  treatment  and  control 
conditions  in  the  use  of  SACD  activities  was  more  than  expected  by  chance  (3  out  of  62  comparisons),  and  in 
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all  cases  they  favored  the  treatment  group.  An  important  point  about  this  difference  between  treatment  and 
control  teachers  is  that  this  training  took  place  just  prior  to  the  start  of  the  school  year.  Because  initial  data 
collection  happened  after  program  implementation  began,  these  differences  may  reflect  the  fact  that  program 
implementation  and  program  training  for  staff  started  before  initial  data  collection. 

Impacts  on  Use  of  SACD  Activities 

The  introduction  of  the  formal  CSP  program  would  be  expected  to  increase  the  use  of  SACD  activities  in  the 
treatment  schools  in  comparison  to  the  control  schools.  The  analysis  of  this  impact  is  based  on  the  Teacher 
Report  on  Classroom  and  School.  Every  spring,  third-,  fourth-,  and  fifth-grade  teachers  provided  information 
through  the  Teacher  Report  on  Classroom  and  School  about  the  social  and  character  development  activities 
they  used  in  their  classrooms.  Specifically,  information  from  the  Teacher  Report  on  Classroom  and  School 
was  used  to  determine  the  difference  between  treatment  and  control  teachers  in  the  following  areas: 

1 . the  use  of  SACD  activities  in  their  classrooms  overall  and  by  SACD  goal; 

2.  the  use  of  materials  and  strategies  to  implement  the  SACD  activities  within  classrooms  and  within 
the  entire  school; 

3.  the  use  of  staff  development  to  support  the  teachers;  and 

4.  teacher  support  for  SACD  efforts  in  the  school  and  the  use  of  practices  conducive  to  the  social  and 
character  development  of  students. 

Teacher  Report  on  Classroom  and  School  consent  and  completion  rates  (table  3.4)  led  to  78  percent  to  95 
percent  of  all  teachers  having  data  for  the  3 years.  To  estimate  intervention  impacts  for  each  of  the  outcomes, 
testing  of  the  statistical  significance  of  the  differences  in  means  was  used.  Preliminary  analysis  indicated  little 
or  no  gains  in  precision  from  using  covariates.  Before  testing  the  mean  differences,  the  data  were  weighted 
such  that  each  school  received  equal  weight.  Standard  errors  of  the  impact  estimates  accounted  for  the 
clustering  of  teachers  within  schools.  In  addition,  a set  of  heuristics  (described  in  chapter  1)  was  applied  to 
determine  whether  each  outcome  domain  was  statistically  significant  after  adjustments  were  made  for  the 
multiple  tests  conducted. 

Use  of  Activities 

The  percentages  of  control  teachers  who  reported  using  any  SACD  activities  in  their  classrooms  ranged  from 
87  percent  to  96  percent  over  the  3 years  (table  3.13,  panel  1).  For  the  six  individual  SACD  goals,  the  ranges 
varied  from  67  percent  to  84  percent  in  Year  1,  67  percent  to  88  percent  in  Year  2,  and  57  percent  to  81 
percent  in  Year  3.  Control  teachers’  use  of  behavior  management  activities  ranged  from  85  percent  to  91 
percent  over  this  period.  The  percentages  of  control  teachers  who  reported  using  any  SACD  activities  in  their 
classrooms  for  at  least  1 hour  per  week  (panel  2)  ranged  from  63  percent  to  82  percent  over  the  3 years.  For 
the  six  individual  SACD  goals,  the  ranges  varied  from  18  percent  to  37  percent  in  Year  1,  20  percent  to  40 
percent  in  Year  2,  and  11  percent  to  37  percent  in  Year  3.  Control  teachers’  use  of  behavior  management 
activities  for  1 hour  per  week  ranged  from  75  percent  to  79  percent  over  this  period.  These  findings  show 
that  the  control  schools  were  using  these  activities  as  part  of  their  standard  practice  related  to  social  and 
character  development. 

For  teachers’  reported  use  of  any  SACD  activity  (panels  1 and  2),  48  comparisons  were  made,  with  2 expected 
to  be  significant  by  chance.  The  percentages  of  treatment  teachers  using  any  SACD  activity,  or  specific  SACD 
activities,  at  any  point  in  the  previous  year  were  not  significantly  different  from  the  percentages  of  control 
teachers  in  any  year.  There  was  a significant  difference  between  treatment  and  control  teacher  reports  of 
using  behavior  management  activities  for  at  least  1 hour  per  week  in  Year  3,  with  more  control  teachers 
reporting  these  activities  (impact  = -14  percentage  points).  After  the  heuristics  were  applied,  the  domain  for 
engagement  in  SACD  activities  showed  CSP  had  a statistically  significant  impact  in  Year  1. 
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For  teachers’  reported  use  of  any  named  SACD  activity  (panels  3 and  4),  42  comparisons  were  made,  with  2 
expected  to  be  significant  by  chance.  Eight  of  the  14  impact  estimates  in  Year  1,  5 of  the  14  in  Year  2,  and  4 
of  the  14  in  Year  3 were  statistically  significant.  Impacts  on  the  use  of  a named  activity  at  any  point  in  the 
previous  year  (panel  3)  were  as  follows:  violence  prevention  and  peace  promotion  in  all  3 years  (impact  = 28, 
30,  and  38  percentage  points),  social  and  emotional  development  in  Year  1 (impact  = 40  percentage  points) 
and  Year  2 (impact  = 44  percentage  points),  character  education  in  Year  2 (impact  = 37  percentage  points) 
and  Year  3 (impact  = 29  percentage  points),  tolerance  and  diversity  in  Year  1 (impact  = 20  percentage  points) 
and  Year  2 (impact  = 33  percentage  points),  and  the  use  of  any  named  activity  in  all  3 years  (impact  = 32,  34, 
and  40  percentage  points).  In  Year  1 there  were  significant  impacts  on  the  use  of  any  named  activity  for  at 
least  1 hour  per  week  (panel  4)  for  violence  prevention  and  peace  promotion  (impact  = 18  percentage 
points),  social  and  emotional  development  (impact  = 21  percentage  points),  and  character  education  (impact 
= 23  percentage  points).  The  impact  on  the  use  of  any  named  activity  for  at  least  1 hour  per  week  was 
significant  in  Year  1 (impact  = 34  percentage  points)  and  Year  3 (impact  = 23  percentage  points).  The  overall 
impact  of  CSP  on  the  domain  for  engagement  in  named  SACD  activities  was  significant  in  all  3 years. 
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Table  3.13.  Impacts  on  teacher-reported  SACD  classroom  activities — CSP 


Panel  1 : Engagement  in  any  activities  to  promote  SACD  goals1 


Year  1 

Year  2 

Year  3 

(Spring  3rd  grade) 

(Spring  4th  grade) 

(Spring  5th  grade) 

SACD  activity 

T reat- 
ment 

Control 

Impact 

p-value 

T reat- 
ment 

Control 

Impact 

p-value 

T reat- 
ment3 

Control 

Impact 

p-value 

Teacher  sample  size 

65.0 

61.0 

57.0 

60.0 

63.0 

62.0 

Violence  prevention  and  peace  promotion 
(percent) 

77.4 

67.1 

10.3 

0.287 

73.0 

72.3 

0.8 

0.935 

72.6 

61.8 

10.8 

0.205 

Social  and  emotional  development 
(percent) 

88.4 

72.5 

16.0 

0.151 

79.0 

73.2 

5.8 

0.358 

72.3 

65.1 

7.2 

0.449 

Character  education  (percent) 

95.2' 

' 84.0 

11.2 

0.097 

91.6 

87.5 

4.1 

0.382 

83.0 

80.6 

2.4 

0.751 

Tolerance  and  diversity  (percent) 

84.9 

74.7 

10.2 

0.420 

73.5 

72.4 

1.1 

0.795 

64.9 

60.0 

4.9 

0.684 

Risk  prevention  and  health  promotion 
(percent) 

76.4 

73.8 

2.6 

0.804 

62.0 

73.3 

-11.2 

0.341 

66.6 

68.8 

-2.2 

0.844 

Civic  responsibility  and  community  service 
(percent) 

72.4 

78.7 

-6.3 

0.650 

65.2 

67.1 

-1.9 

0.819 

58.7 

56.5 

2.2 

0.861 

Any  SACD  goal  (percent) 

96.7 

90.8 

5.9 

0.288 

94.3 

96.3 

-2.0 

0.550 

94.1 

87.3 

6.8 

0.283 

Behavior  management  (percent) 

96.9 

90.6 

6.3 

0.144 

89.2 

85.2 

3.9 

0.565 

80.3 

88.6 

-8.3 

0.110 

See  notes  at  end  of  table. 
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Table  3.13.  Impacts  on  teacher-reported  SACD  classroom  activities — CSP — Continued 


Panel  2:  Engagement  in  any  activities  to  promote  SACD  goals  for  at  least  1 hour  per  week 


Year  1 

Year  2 

Year  3 

(Spring  3rd  grade) 

(Spring  4th  grade) 

(Spring  5th  grade) 

SACD  activity 

T reat- 
ment 

Control 

Impact 

p-value 

T reat- 
ment 

Control 

Impact  p-value 

T reat- 
ment3 

Control 

Impact 

p-value 

Teacher  sample  size 

65.0 

61.0 

57.0 

60.0 

63.0 

62.0 

Violence  prevention  and  peace  promotion 
(percent) 

28.5 

29.6 

-1.0 

0.932 

24.9 

28.8 

-3.9 

0.623 

26.1 

27.4 

-1.3 

0.867 

Social  and  emotional  development 
(percent) 

39. 2A 

26.7 

12.6 

0.087 

32.5 

27.5 

4.9 

0.654 

21.6 

24.9 

-3.3 

0.748 

Character  education  (percent) 

55. 2A 

36.9 

18.2 

0.098 

48.6 

39.6 

9.0 

0.411 

31.3 

37.3 

-6.0 

0.547 

Tolerance  and  diversity  (percent) 

27.1 

18.0 

9.0 

0.363 

24.5 

30.9 

-6.3 

0.496 

27.1 

11.4 

15.7 

0.145 

Risk  prevention  and  health  promotion 
(percent) 

35.6 

24.7 

10.9 

0.427 

30.1 

29.0 

1.0 

0.904 

24.0 

29.3 

-5.3 

0.659 

Civic  responsibility  and  community  service 
(percent) 

27.4 

28.3 

-1.0 

0.933 

18.2 

20.1 

-1.8 

0.876 

31.8 

15.0 

16.8 

0.133 

Any  SACD  goal  (percent) 

68.8 

62.8 

6.0 

0.431 

76.2 

78.1 

-1.8 

0.824 

76.8 

81.5 

-4.7 

0.373 

Behavior  management  (percent) 

80.6 

78.8 

1.8 

0.789 

70.8 

74.6 

-3.8 

0.720 

61.3* 

' 75.3 

-14.0 

0.046 

See  notes  at  end  of  table. 
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Table  3.13.  Impacts  on  teacher-reported  SACD  classroom  activities — CSP — Continued 


Panel  3:  Engagement  in  activities  to  promote  SACD  goals  linked  to  named  SACD  programs2 


Year  1 Year  2 Year  3 

(Spring  3rd  grade) (Spring  4th  grade) (Spring  5th  grade) 


SACD  activity 

T reat- 
ment 

Control 

Impact 

p-value 

T reat- 
ment 

Control 

Impact 

p-value 

T reat- 
ment3 

Control 

Impact 

p-value 

Teacher  sample  size 

65.0 

61.0 

57.0 

60.0 

63.0 

62.0 

Violence  prevention  and  peace  promotion 
(percent) 

38.8* 

10.5 

28.3 

0.022 

50.4* 

20.5 

29.9 

0.016 

51.5* 

13.3 

38.2 

0.003 

Social  and  emotional  development 
(percent) 

r 

t 

39.7 

0.003 

54.5* 

10.3 

44.3 

0.007 

47.0 

0.0 

47.0 

t 

Character  education  (percent) 

35. 8A 

7.1 

28.7 

0.063 

49.0* 

11.7 

37.3 

0.001 

33.7* 

4.9 

28.8 

0.018 

Tolerance  and  diversity  (percent) 

t* 

t 

19.6 

0.015 

t* 

t 

32.5 

0.014 

15.6 

0.0 

15.6 

t 

Risk  prevention  and  health  promotion 
(percent) 

36.8 

23.0 

13.8 

0.381 

28.6 

23.8 

4.8 

0.569 

26.0 

18.9 

7.1 

0.545 

Civic  responsibility  and  community  service 
(percent) 

t 

t 

1.5 

t 

9.5 

8.9 

0.6 

0.936 

6.3 

5.6 

0.7 

0.811 

Any  SACD  goal  (percent) 

67.8* 

35.7 

32.1 

0.025 

74.1* 

39.8 

34.3 

0.005 

69.7* 

29.9 

39.8 

0.007 

See  notes  at  end  of  table. 
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Table  3.13.  Impacts  on  teacher-reported  SACD  classroom  activities — CSP — Continued 

Panel  4:  Engagement  in  activities  to  promote  SACD  goals  linked  to  named  SACD  programs  for  at  least  1 hour  per  week 


Year  1 Year  2 Year  3 

(Spring  3rd  grade) (Spring  4th  grade) (Spring  5th  grade) 


SACD  activity 

T reat- 
ment 

Control 

Impact 

p-value 

T reat- 
ment 

Control 

Impact 

p-value 

T reat- 
ment3 

Control 

Impact 

p-value 

Teacher  sample  size 

65.0 

61.0 

57.0 

60.0 

63.0 

62.0 

Violence  prevention  and  peace  promotion 
(percent) 

r 

t 

17.8 

0.009 

21.1 

14.4 

6.7 

0.334 

19.9 

11.8 

8.0 

0.211 

Social  and  emotional  development 
(percent) 

r 

t 

21.4 

0.012 

23.9 

5.8 

18.1 

0.145 

13.5 

0.0 

13.5 

t 

Character  education  (percent) 

r 

t 

23.4 

0.005 

28. 0A 

10.9 

17.1 

0.055 

tA 

t 

8.3 

0.097 

Tolerance  and  diversity  (percent) 

tA 

t 

8.3 

0.057 

16.2 

0.0 

16.2 

t 

8.4 

0.0 

8.4 

t 

Risk  prevention  and  health  promotion 
(percent) 

18.1 

6.4 

11.8 

0.157 

17.0 

18.6 

-1.6 

0.834 

13.5 

11.8 

1.7 

0.860 

Civic  responsibility  and  community  service 
(percent) 

t 

t 

1.7 

t 

i 

i 

-2.9 

0.664 

0.0 

0.0 

0.0 

t 

Any  SACD  goal  (percent) 

45.1* 

10.8 

34.3 

0.008 

36. 8A 

25.5 

11.3 

0.097 

38.7* 

16.0 

22.7 

0.047 

t Not  applicable. 

$ Reporting  standards  not  met.  Values  suppressed  to  protect  confidentiality. 

* Treatment  group  significantly  different  from  control  group  at  the  .05  level. 

A T reatment  group  significantly  different  from  control  group  at  the  . 1 0 to  > .05  level. 

1 1n  Year  1 , the  omnibus  impact  for  all  the  outcomes  measured  together  was  positive  and  statistically  significant  on  the  basis  of  a multivariate  statistical  test. 

2 In  Year  1 , based  on  univariate  statistical  tests,  at  least  half  of  the  impacts  were  positive  and  statistically  significant  and  no  impact  was  negative  and  statistically  significant.  In  all  3 
years,  at  least  one  outcome  remained  positive  and  statistically  significant  and  no  outcome  was  negative  and  statistically  significant  after  applying  the  Benjamini-Hochberg  (1995) 
procedure  to  adjust  significance  levels  downward  to  account  for  the  multiple  testing  of  impacts. 

3 One  of  the  treatment  group  schools  that  had  been  a K-5  school  in  prior  years  became  a K-4  school  and  continued  implementation  of  the  intervention.  The  Cohort  1 students,  who 
were  in  fifth  grade  during  the  2006-07  school  year,  went  to  an  intermediate  school  that  had  fifth  and  sixth  grades.  The  intermediate  school  did  not  implement  the  intervention.  These 
fifth-grade  students,  their  primary  caregivers,  and  their  teachers  were  included  in  the  spring  2007  data  collection  and  all  analyses  based  upon  spring  2007  data. 

NOTE:  Weights,  which  assign  equal  weight  to  each  school  within  the  program,  were  used  in  producing  the  treatment,  control,  and  overall  means. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Use  of  Materials  and  Strategies 

For  use  of  materials  and  strategies  to  support  SACD  goals,  87  comparisons  were  made,  with  4 expected  to  be 
significant  by  chance.  Several  significant  impacts  were  found  on  treatment  teachers’  use  of  materials  and 
strategies  in  all  3 years.  In  Year  1,  more  treatment  teachers  used  role-playing  (impact  = 20  percentage  points), 
direct  instruction  of  SACD  (impact  =16  percentage  points),  skill  training  (impact  = 33  percentage  points), 
and  guided  visualization  (impact  = 36  percentage  points)  (table  3.14).  In  Year  2,  more  treatment  teachers 
continued  to  use  direct  instruction  of  SACD  (impact  =18  percentage  points),  and  more  treatment  teachers 
used  teacher  guides  (impact  =19  percentage  points)  and  student  materials  (impact  =16  percentage  points). 
In  Year  3,  more  treatment  teachers  used  teacher  guides  (impact  = 21  percentage  points).  The  average  number 
of  strategies  used  was  significantly  different  in  Year  1 (by  1.6  activities  on  average).  The  impact  on  the  domain 
of  materials  and  strategies  was  not  statistically  significant  in  any  of  the  3 years.  Regarding  the  use  of 
schoolwide  strategies,  18  comparisons  were  made  between  treatment  and  control  teacher  reports,  with  1 
expected  to  be  significant  by  chance.  There  were  no  statistically  significant  differences  between  treatment  and 
control  teacher  reports  on  use  of  schoolwide  strategies  in  any  year  (these  data  are  not  shown  in  a table). 
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Table  3.14.  Impacts  on  use  of  SACD  classroom  materials  and  teaching  strategies — CSP 


Year  1 Year  2 Year  3 

(Spring  3rd  grade) (Spring  4th  grade) (Spring  5th  grade) 


SACD  material  and  teaching  strategy 

Treat- 

ment 

Control 

Impact 

p-value 

T reat- 
ment 

Control 

Impact 

p-value 

T reat- 
ment1 

Control 

Impact 

p-value 

Teacher  sample  size 

65 

61 

57 

60 

63 

62 

Use  of  SACD  materials  (percent) 

Teacher  guides  (manuals,  curricula) 

87.1 

69.7 

17.4 

0.109 

88.0* 

69.4 

18.6 

0.002 

84.1* 

62.8 

21.3 

0.015 

Student  materials  (workbooks  or  sheets) 

78.4 

70.7 

7.6 

0.405 

87.7* 

71.8 

15.9 

0.004 

74.6 

65.7 

9.0 

0.221 

Instructional  aids  (games,  software,  videos) 

49.2 

53.9 

-4.7 

0.690 

54.0 

46.0 

7.9 

0.469 

60. 5A 

40.4 

20.1 

0.096 

Giveaways  (bookmarks,  stickers) 

48.1 

72.5 

-24.4 

0.136 

51.1 

52.1 

-1.0 

0.940 

46.4 

60.2 

-13.9 

0.187 

Children’s  literature 

69.6 

48.2 

21.4 

0.102 

68.9 

53.8 

15.1 

0.119 

48.7 

45.6 

3.1 

0.637 

Other  types  of  materials 

5.3 

6.2 

-0.9 

0.857 

t 

t 

-5.5 

0.296 

10.9 

11.6 

-0.7 

0.915 

Did  not  use  any  of  these  materials 

t 

t 

-6.6 

0.206 

t 

t 

-2.5 

0.520 

t 

t 

-8.6 

0.111 

Use  of  teaching  strategies  (percent) 

Role-playing 

83.0* 

62.6 

20.4 

0.034 

85.7 

72.3 

13.5 

0.111 

87.5 

79.9 

7.6 

0.282 

Cooperative  learning 

91. 2A 

98.8 

-7.6 

0.083 

96.0 

98.9 

-2.9 

0.294 

100.0 

98.7 

1.3 

t 

Peer  group  discussions 

94. 2A 

82.8 

11.4 

0.064 

93.2 

88.5 

4.7 

0.249 

98.3 

94.1 

4.2 

0.255 

Direct  instruction  of  SACD 

98.3* 

82.9 

15.5 

0.043 

97.0* 

79.5 

17.6 

0.013 

99.0 

95.0 

4.1 

0.181 

Skill  training 

69.0* 

36.2 

32.8 

0.023 

53.1 

40.9 

12.1 

0.344 

80.2 

84.3 

-4.1 

0.533 

Incorporating  SACD  into  academic 
curriculum 

90.6 

78.1 

12.5 

0.248 

88. 1A 

74.5 

13.6 

0.082 

96.3 

93.5 

2.8 

0.492 

Parent  training 

10.2 

12.8 

-2.6 

0.700 

t 

t 

-5.2 

0.146 

39.6 

30.4 

9.3 

0.367 

Parent/community  involvement 

25.4 

34.5 

-9.1 

0.397 

31.5 

23.1 

8.4 

0.424 

51.7 

46.9 

4.7 

0.600 

Mentoring 

59.5 

48.2 

11.3 

0.293 

53.4 

56.7 

-3.2 

0.863 

84. 6A 

67.7 

16.9 

0.080 

Good  behavior  notes  sent  home  daily 
or  weekly 

80.3 

92.2 

-11.9 

0.308 

86.9 

93.0 

-6.1 

0.128 

96.3 

97.5 

-1.2 

0.734 

Presenting  role  models 

73.8 

79.1 

-5.3 

0.596 

75.7 

67.9 

7.8 

0.472 

85.8 

91.0 

-5.3 

0.394 

See  notes  at  end  of  table. 
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Table  3.14.  Impacts  on  use  of  SACD  classroom  materials  and  teaching  strategies — CSP — Continued 


Year  1 

(Spring  3rd  grade) 

Year  2 

(Spring  4th  grade) 

Year  3 

(Spring  5th  grade) 

SACD  material  and  teaching  strategy 

T reat- 
ment 

Control 

Impact 

p-value 

T reat- 
ment 

Control 

Impact 

p-value 

T reat- 
ment1 

Control 

Impact 

p-value 

Use  of  teaching  strategies  (percent) — 
Continued 

Targeted  story  reading  or  writing  on  social 
and  character  development  themes 

86.2 

75.8 

10.4 

0.186 

95.8 

85.4 

10.3 

0.104 

98.0 

93.3 

4.7 

0.278 

Peer  mediation 

60.3 

40.9 

19.4 

0.288 

60.0 

59.7 

0.3 

0.975 

72.2 

68.0 

4.2 

0.738 

Honor  roll  for  positive  behavior 

69.1 

70.5 

-1.4 

0.889 

75.8 

84.7 

-8.9 

0.311 

81.5 

79.8 

1.7 

0.843 

Pledges  or  recitations  on  social  and 
character  development  themes 

51.6 

33.7 

17.8 

0.228 

49.6 

48.8 

0.8 

0.962 

74.8 

65.7 

9.1 

0.578 

Guided  visualization 

74.9* 

38.5 

36.4 

0.004 

58.5 

55.0 

3.5 

0.673 

75.3 

59.0 

16.2 

0.134 

Student-led/student-assisted  instruction 

67.2 

50.9 

16.2 

0.198 

57.8 

46.2 

11.6 

0.245 

78.7 

75.2 

3.5 

0.697 

Journaling 

66.8 

76.0 

-9.1 

0.469 

61.0 

62.7 

-1.7 

0.881 

86.2 

81.1 

5.1 

0.484 

Time  out  for  negative  behavior 

93.7 

93.0 

0.7 

0.905 

87.9 

88.2 

-0.3 

0.971 

93.2 

93.3 

-0.1 

0.985 

Daily  or  weekly  rewards  for  positive 
behavior 

94.4 

87.9 

6.5 

0.453 

96.4 

97.0 

-0.6 

0.836 

100.0 

96.3 

3.7 

t 

Any  strategy 

100.0 

100.0 

0.0 

t 

100.0 

100.0 

0.0 

t 

100.0 

100.0 

0.0 

t 

Number  of  strategies  (mean) 

14.2* 

12.6 

1.6 

0.029 

13.9 

13.3 

0.6 

0.296 

16.5 

15.5 

1.0 

0.278 

t Not  applicable. 

$ Reporting  standards  not  met.  Values  suppressed  to  protect  confidentiality. 

* Treatment  group  significantly  different  from  control  group  at  the  .05  level. 

A Treatment  group  significantly  different  from  control  group  at  the  .10  to  > .05  level. 

1 One  of  the  treatment  group  schools  that  had  been  a K-5  school  in  prior  years  became  a K-4  school  and  continued  implementation  of  the  intervention.  The  Cohort  1 students,  who 
were  in  fifth  grade  during  the  2006-07  school  year,  went  to  an  intermediate  school  that  had  fifth  and  sixth  grades.  The  intermediate  school  did  not  implement  the  intervention.  These 
fifth-grade  students,  their  primary  caregivers,  and  their  teachers  were  included  in  the  spring  2007  data  collection  and  all  analyses  based  upon  spring  2007  data. 

NOTE:  Weights,  which  assign  equal  weight  to  each  school  within  the  program,  were  used  in  producing  the  treatment,  control,  and  overall  means. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Participation  in  Professional  Development 

Regarding  reported  participation  in  professional  development,  27  comparisons  were  made  over  3 years,  with 
2 expected  to  be  significant  by  chance.  In  Year  1,  CSP  had  a statistically  significant  effect  on  treatment 
teachers’  participation  in  professional  development,  with  more  treatment  teachers  reporting  SACD  training  in 
the  past  12  months  (impact  = 39  percentage  points)  and  more  hours  of  training  (by  5 hours  on  average).  In 
terms  of  specific  SACD  goals,  more  treatment  teachers  reported  training  in  character  education  (impact  = 38 
percentage  points)  and  behavior  management  (impact  = 16.3  percentage  points).  In  Year  3,  there  was  a 
significant  impact  on  teachers’  training  in  violence  prevention  and  peace  promotion  (impact  =13  percentage 
points).  CSP  had  a significant  impact  on  the  domain  of  professional  development  in  Year  1. 
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Table  3.15.  Impacts  on  teacher-reported  SACD  professional  development — CSP 


SACD  professional  development1 

Year  1 

(Spring  3rd  grade) 

Year  2 

(Spring  4th  grade) 

Year  3 

(Spring  5th  grade) 

T reat- 
ment 

Control  Impact 

p-value 

T reat- 
ment 

Control  Impact 

p-value 

T reat- 
ment 

Control  Impact 

p-value 

Teacher  sample  size 

65 

61 

57 

60 

63 

62 

SACD  training  in  past  12  months  (percent) 

82.0* 

43.0 

39.0 

0.009 

54.9 

51.1 

3.8 

0.723 

56.1 

49.3 

6.8 

0.524 

Hours  of  SACD  training  (mean) 

7.9* 

2.9 

5.0 

0.006 

3.8 

6.2 

-2.4 

0.375 

4.9 

4.6 

0.3 

0.899 

Training  by  goal  (percent) 

Violence  prevention  and  peace  promotion 

21.0 

8.2 

12.9 

0.143 

8.4 

10.8 

-2.4 

0.753 

t* 

t 

13.1 

0.036 

Social  and  emotional  development 

29.4 

9.3 

20.1 

0.118 

18.4 

18.6 

-0.2 

0.978 

13.3 

9.7 

3.6 

0.561 

Character  education 

57.6* 

19.7 

37.9 

0.016 

32.7 

16.2 

16.5 

0.106 

25.8 

12.9 

13.0 

0.122 

Tolerance  and  diversity 

23. 5A 

6.9 

16.6 

0.071 

8.1 

18.2 

-10.2 

0.273 

21.0 

12.9 

8.1 

0.231 

Risk  prevention  and  health  promotion 

14.1 

8.2 

5.9 

0.387 

8.1 

20.1 

-12.0 

0.171 

16.9 

11.7 

5.3 

0.333 

Civic  responsibility  and  community  service 

15.1 

19.7 

-4.7 

0.601 

i 

t 

1.8 

0.566 

6.4 

0.0 

6.4 

t 

Behavior  management 

35.6* 

19.3 

16.3 

0.021 

27.8 

26.2 

1.6 

0.904 

31.3 

22.2 

9.1 

0.351 

t Not  applicable. 

$ Reporting  standards  not  met.  Values  suppressed  to  protect  confidentiality. 

* Treatment  group  significantly  different  from  control  group  at  the  .05  level. 

A T reatment  group  significantly  different  from  control  group  at  the  . 1 0 to  > .05  level. 

1 1n  Year  1 , at  least  one  outcome  remained  positive  and  statistically  significant  and  no  outcome  was  negative  and  statistically  significant  after  applying  the  Benjamini-Hochberg  (1995) 
procedure  to  adjust  significance  levels  downward  to  account  for  the  multiple  testing  of  impacts. 

2 One  of  the  treatment  group  schools  that  had  been  a K-5  school  in  prior  years  became  a K-4  school  and  continued  implementation  of  the  intervention.  The  Cohort  1 students,  who 
were  in  fifth  grade  during  the  2006-07  school  year,  went  to  an  intermediate  school  that  had  fifth  and  sixth  grades.  The  intermediate  school  did  not  implement  the  intervention.  These 
fifth-grade  students,  their  primary  caregivers,  and  their  teachers  were  included  in  the  spring  2007  data  collection  and  all  analyses  based  upon  spring  2007  data. 

NOTE:  Weights,  which  assign  equal  weight  to  each  school  within  the  program,  were  used  in  producing  the  treatment,  control,  and  overall  means. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Attitudes  and  Practices 

Teachers  reported  on  their  enthusiasm  for  SACD  efforts  in  their  schools  (these  data  are  not  shown  in  a table) 
by  indicating  enthusiasm,  cooperation,  or  open  dislike.  They  also  reported  on  the  SACD  practices  of  teachers 
and  staff  members  in  their  school  (these  data  are  not  shown  in  a table).  These  practices  included  modeling 
positive  character  and  behavior  traits  with  students  and  fellow  teachers,  involving  students  in  making 
decisions,  giving  students  a voice  in  school  governance,  the  school  encouraging  parent  involvement  in 
children’s  social  and  character  development,  and  using  developmentally  appropriate  discipline  strategies 
rather  than  punishment  for  misbehavior.  Twenty-seven  comparisons  were  made  over  3 years,  with  1 expected 
to  be  significant  by  chance.  There  were  no  statistically  significant  estimated  impacts  on  teachers’  enthusiasm 
for  SACD  efforts  in  their  schools  in  any  of  the  years.  There  were  2 significant  differences  in  the  treatment 
and  control  teacher  reports  of  the  overall  use  of  practices  conducive  to  students’  social  and  character 
development  (out  of  6 comparisons  made  in  each  year,  or  18  overall);  in  Year  1,  fewer  treatment  teachers 
reported  modeling  positive  traits  with  students  (impact  = -22  percentage  points),  and  in  Year  3 more 
treatment  teachers  reported  using  discipline  strategies  that  focused  on  promoting  development  rather  than 
simply  punishing  for  misbehavior  (impact  = 25  percentage  points).  There  was  no  significant  impact  on  the 
domain  in  any  year. 

Year-by-Year  Impacts  on  Students  and  Perceptions  of  School  Climate 

The  primary  research  question  for  the  CSP  evaluation  was  this: 

What  is  the  average  effect  of  CSP  on  children’s  social  and  emotional  competence,  behavior, 
academics,  and  perceptions  of  school  climate? 

The  first  approach  to  answering  this  question  was  to  examine  the  year-by-year  impacts  of  CSP  on  these 
student  and  school  climate  outcomes  over  the  3 years  as  the  students  progressed  from  third  through  fifth 
grades. 

Equation  (2)  (described  in  chapter  1)  was  estimated  to  provide  CSP  impacts  on  the  20  outcomes  using  data 
from  the  10  treatment  and  control  schools.  For  the  CSP  evaluation,  equation  (2)  excluded  the  program  fixed 
effects  (Op)  and  included  program-specific  covariates  and  random  school  effects  covariates.  Table  3.16  lists 
the  covariates  used  with  outcomes  from  each  report  in  the  CSP  analysis. 
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Table  3.16.  Covariates  used  with  outcomes  from  each  report  for  analysis — CSP 


Potential  covariate 

CR 

outcome 

PCR 

outcome 

TRS 

outcome 

TRCS 

outcome 

Total  number 

17 

24 

25 

9 

Child-reported 

Female 

■/ 

S 

■/ 

Hispanic 

>4 

S 

■/ 

Black  (non-Hispanic) 

■/ 

s 

■/ 

Other  ethnicity 

■/ 

s 

■/ 

Age  in  years 

■/ 

s 

■/ 

Scales 

Afraid  at  School 

Altruistic  Behavior 

■/ 

s 

■/ 

Empathy 

Engagement  with  Learning 

■/ 

■/ 

Negative  School  Orientation 

■/ 

s 

Normative  Beliefs  About  Aggression 

s 

Sense  of  School  as  a Community 

■/ 

Problem  Behavior 

Self-Efficacy  for  Peer  Interactions 

s 

Victimization  at  School 

■/ 

■/ 

Primary  caregiver-reported 

Age  in  years 

■/ 

Completed  high  school  or  eguivalent 

■/ 

s 

>4 

Some  college 

■/ 

s 

>4 

Bachelor’s  or  higher  degree 

■/ 

s 

>4 

Highest  level  of  education  in  household 

Completed  high  school  or  eguivalent 

■/ 

s 

■/ 

Some  college 

■/ 

s 

■/ 

Bachelor’s  or  higher  degree 

■/ 

s 

>4 

Mother  present  in  home  life 

Mother  and  father  present 

s 

Respondent  someone  other  than  mother  or  father 

s 

Number  of  people  in  household 

Household  income:  $20,000  to  $40,000 

s 

■/ 

Household  income:  $40,000  to  $60,000 

s 

■/ 

Household  income:  More  than  $60,000 

s 

■/ 

Income-to-poverty-threshold  ratio:  Below  135  percent 

Income-to-poverty-threshold  ratio:  135  to  185  percent 

Full-time  employment 

✓ 

■/ 

Part-time  employment 

■/ 

■/ 

See  notes  at  end  of  table. 
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Table  3.16.  Covariates  used  with  outcomes  from  each  report  for  analysis — CSP — Continued 


Potential  covariate 

CR 

outcome 

PCR 

outcome 

TRS 

outcome 

TRCS 

outcome 

Parental  scales 

APQ-Poor  Monitoring  and  Supervision  Subscale 

APQ-Positive  Parenting  Subscale 

Child-Centered  Social  Control 

Y 

Confusion,  Hubbub,  and  Order 

Y 

Community  Resources 

Community  Risk 

Parent  and  Teacher  Involvement 

Child  scales 

Altruistic  Behavior 

Positive  Social  Behavior 

Y 

Problem  Behavior 

Teacher-reported 

Female 

Y 

Hispanic 

Y 

Black  (non-Hispanic) 

Y 

Other  ethnicity 

Y 

Total  teaching  experience 

Y 

Total  experience  in  current  school 

Y 

Regular  certificate 

Y 

Other  certificate 

Y 

Highest  degree-bachelor’s 

Y 

Child  scales 

Academic  Competence  and  Motivation 

Y 

ADHD-Related  Behavior 

Y 

Altruistic  Behavior 

Y 

Positive  Social  Behavior 

Problem  Behavior 

Y 

Parent  and  Teacher  Involvement 

NOTE:  Abbreviations  are 
CR:  Child  Report 
PCR:  Primary  Caregiver  Report 
TRS:  Teacher  Report  on  Student 
TRCS:  Teacher  Report  on  Classroom  and  School 
ADHD:  Attention  deficit  hyperactivity  disorder 
APQ:  Alabama  Parenting  Questionnaire 
Y : Covariate  used 
Blank  cell:  Covariate  not  used 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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To  assess  statistical  power  of  the  program -level  impact  estimates,  minimum  detectable  impacts  in  effect  size 
units  (MDES)  for  each  outcome  measure  were  calculated  for  the  CSP  evaluation  (table  3.17).  MDES 
represent  the  smallest  impacts  in  effect  size  (standard  deviation)  units  that  can  be  detected  with  a high 
probability  (80%).  MDES  are  primarily  a function  of  study  sample  sizes,  the  degrees  of  freedom  available  for 
statistical  tests,  and  design  effects  due  to  clustering  (Schochet  2005).  For  the  CSP  evaluation,  the  MDES 
ranged  from  0.113  to  0.57  for  the  child -level  outcomes  based  on  the  Child,  Caregiver,  and  Teacher  Report  on 
Student  and  from  0.258  to  0.625  for  the  school  climate  outcomes  based  on  the  Teacher  Report  on  Classroom 
and  School.  In  general,  the  MDES  for  the  school  climate  outcomes  were  larger  than  those  for  the  child-level 
outcomes. 


245 


Chapter  3.  Competence  Support  Program 


Table  3.17.  Adjusted  minimum  detectable  effect  sizes  for  impact  evaluation — CSP 


Outcome  measure-Report 

Year  1 

Year  2 

Year  3 

Social  and  Emotional  Competence  Domain 

Self-Efficacy  for  Peer  Interaction-CR 

0.129 

0.184 

0.116 

Normative  Beliefs  About  Aggression-CR 

0.113 

0.154 

0.161 

Empathy-CR 

0.238 

0.114 

0.182 

Behavior  Domain 

Altruistic  Behavior-CR 

0.146 

0.202 

0.171 

Altruistic  Behavior-PCR 

0.216 

0.239 

0.169 

Altruistic  Behavior-TRS 

0.185 

0.436 

0.570 

Positive  Social  Behavior-PCR 

0.172 

0.172 

0.170 

Positive  Social  Behavior-TRS 

0.310 

0.301 

0.247 

Problem  Behavior-CR 

0.182 

0.334 

0.168 

Problem  Behavior-PCR 

0.210 

0.136 

0.226 

Problem  Behavior-TRS 

0.352 

0.304 

0.369 

ADHD-Related  Behavior-TRS 

0.290 

0.206 

0.250 

Academics  Domain 

Engagement  with  Learning-CR 

0.117 

0.173 

0.201 

Academic  Competence  and  Motivation-TRS 

0.194 

0.142 

0.118 

Perceptions  of  School  Climate  Domain 

Positive  School  Orientation-CR 

0.113 

0.310 

0.371 

Negative  School  Orientation-CR 

0.162 

0.251 

0.354 

Student  Afraid  at  School-CR 

0.135 

0.222 

0.215 

Victimization  at  School-CR 

0.210 

0.237 

0.284 

Feelings  of  Safety-TRCS 

0.541 

0.447 

0.562 

Student  Support  for  Teachers-TRCS 

0.625 

0.309 

0.258 

NOTE:  Abbreviations  are 
CR:  Child  Report 
PCR:  Primary  Caregiver  Report 
TRS:  Teacher  Report  on  Student 
TRCS:  Teacher  Report  on  Classroom  and  School 
ADHD:  Attention  deficit  hyperactivity  disorder 
The  minimum  detectable  effect  (MDE)  formula  used  in  the  calculations  is  as  follows: 


MDE  = f actor (df)  * 


Pi 


(-  + -)  + (l-Pi)  (— + — ) 
\sT  scJ  \sTnT  srnr) 


sTnT  scnc) 


where  Sj-and  sc  are  the  number  of  treatment  and  comparison  schools;  nT  and  nc  are  the  average  number  of  students  per  classroom; 
pi  is  the  intraclass  correlation  (ICC)  at  the  school  level;  and  factor(df)  is  a constant  that  depends  on  the  number  of  degrees  of 
freedom  (df)  available  for  analysis  (and  is  2.802  for  the  pooled  analysis). 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Table  3.18  provides  the  estimates  of  CSP’s  impacts  on  each  of  the  20  outcomes  over  each  of  the  3 years  (60 
impacts  in  total,  with  3 expected  to  be  statistically  significant  by  chance).  Of  the  60  results,  1 was  statistically 
significant.  A beneficial  impact  was  found  on  fourth-graders’  Problem  Behavior  (Primary  Caregiver  Report, 
effect  size  [ES]  = —0.21).  Substantively  important  but  nonsignificant  beneficial  impacts  were  found  in  Year  2 
for  Altruistic  Behavior  (Teacher  Report  on  Student,  ES  = 0.47)  and  Student  Afraid  at  School  (Child  Report, 
ES  = -0.26).  Substantively  important  but  nonsignificant  detrimental  impacts  were  found  on  fifth-graders’ 
Altruistic  Behavior  (Teacher  Report  on  Student,  ES  = -0.41),  Positive  Social  Behavior  (Primary  Caregiver 
Report,  ES  = -0.25),  and  Feelings  of  Safety  (Teacher  Report  on  Classroom  and  School,  ES  = -0.36). 
Application  of  the  heuristics  to  adjust  for  multiple  comparisons  within  each  outcome  domain  indicated  that 
CSP  had  a significant  detrimental  impact  on  the  domain  of  Behavior  in  Year  1 and  a significant  detrimental 
impact  on  the  domain  of  Perceptions  of  School  Climate  in  Year  3. 
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Table  3.18.  Impacts  on  child  and  school  outcomes — CSP 


Year  1 Year  2 Year  3 

(Spring  3rd  grade) (Spring  4th  grade) (Spring  5th  grade) 


T reat-  Effect  T reat-  Effect  T reat-  Effect 


Scale-Report 

ment 

Control 

size 

p-value 

ment 

Control 

size 

p-value 

ment 

Control 

size 

p-value 

Social  and  Emotional  Competence  Domain 

Self-Efficacy  for  Peer  Interactions-CR  (+) 

3.04 

2.97 

0.12 

0.199 

3.15 

3.15 

0.00 

0.978 

3.19 

3.22 

-0.05 

0.581 

Normative  Beliefs  About  Aggression-CR  (-) 

1.30 

1.26 

0.07 

0.401 

1.35 

1.29 

0.12 

0.273 

1.48 

1.52 

-0.05 

0.624 

Empathy-CR  (+) 

2.34 

2.26 

0.17 

0.168 

2.18 

2.23 

-0.11 

0.206 

2.07 

2.07 

-0.02 

0.888 

Behavior  Domain2 

Altruistic  Behavior-CR  (+) 

1.21 

1.30 

-0.11 

0.221 

1.06 

1.00 

0.08 

0.505 

0.99 

1.03 

-0.06 

0.606 

Altruistic  Behavior-PCR  (+) 

2.29 

2.25 

0.04 

0.773 

2.33 

2.31 

0.03 

0.849 

2.25 

2.28 

-0.04 

0.733 

Altruistic  Behavior-TRS  (+) 

1.42 

1.41 

0.04 

0.713 

1.39 

1.25 

0.47 c 

0.132 

1.31 

1.55 

-0.41  0 

0.132 

Positive  Social  Behavior-PCR  (+) 

3.04 

3.01 

0.06 

0.515 

3.11 

3.02 

0.17 

0.125 

3.03A 

3.16 

-0.25° 

0.056 

Positive  Social  Behavior-TRS  (+) 

2.95 

3.05 

-0.16 

0.241 

3.02 

2.93 

0.12 

0.426 

3.09 

3.01 

0.12 

0.386 

Problem  Behavior-CR  (-) 

0.27 

0.25 

0.04 

0.671 

0.32 

0.26 

0.15 

0.428 

0.52 

0.51 

0.01 

0.929 

Problem  Behavior-PCR  (-) 

1.54 

1.57 

-0.11 

0.330 

1.50* 

1.56 

-0.21 

0.042 

1.65 

1.50 

0.20 

0.218 

Problem  Behavior-TRS  (-) 

1.42 

1.38 

0.09 

0.566 

1.48 

1.50 

-0.03 

0.852 

1.48 

1.44 

0.07 

0.712 

ADHD-Related  Behavior-TRS  (-) 

1.73 

1.74 

-0.03 

0.826 

1.70 

1.74 

-0.06 

0.585 

1.71 

1.74 

-0.04 

0.764 

Academics  Domain 

Engagement  with  Learning-CR  (+) 

3.70 

3.68 

0.04 

0.645 

3.71 

3.71 

-0.01 

0.952 

3.61 

3.58 

0.04 

0.745 

Academic  Competence  and  Motivation- 

TRS  (+) 

3.05 

2.97 

0.08 

0.245 

2.96 

2.93 

0.04 

0.622 

2.99 

2.96 

0.03 

0.680 

See  notes  at  end  of  table. 
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Table  3.18.  Impacts  on  child  and  school  outcomes — CSP — Continued 


Year  1 Year  2 Year  3 

(Spring  3rd  grade) (Spring  4th  grade) (Spring  5th  grade) 


T reat-  Effect  T reat-  Effect  T reat-  Effect 


Scale-Report 

ment 

Control 

size 

p-value 

ment 

Control 

size 

p-value 

ment 

Control 

size 

p-value 

Perceptions  of  School  Climate  Domain3 
Positive  School  Orientation-CR  (+) 

2.81 A 

2.71 

0.15 

0.081 

2.64 

2.65 

-0.02 

0.922 

2.37 

2.46 

-0.13 

0.527 

Negative  School  Orientation-CR  (-) 

1.90 

1.96 

-0.11 

0.274 

1.95 

1.98 

-0.05 

0.699 

2.13 

2.19 

-0.10 

0.590 

Student  Afraid  at  School-CR  (-) 

2.26 

2.40 

-0.15 

0.103 

2.1 7A 

2.38 

-0.26° 

0.090 

2.34 

2.38 

-0.04 

0.728 

Victimization  at  School-CR  (-) 

0.84 

0.78 

0.08 

0.529 

0.69 

0.70 

-0.01 

0.947 

0.74 

0.75 

-0.02 

0.896 

Feelings  of  Safety-TRCS  (+) 

3.42 

3.44 

-0.03 

0.927 

3.60 

3.64 

-0.05 

0.834 

3.35 

3.58 

-0.36° 

0.246 

Student  Support  for  Teachers-TRCS  (+) 

3.53 

3.56 

-0.05 

0.890 

3.55 

3.48 

0.10 

0.551 

3.37 

3.45 

-0.11 

0.550 

* Treatment  group  significantly  different  from  control  group  at  the  .05  level. 

A Treatment  group  significantly  different  from  control  group  at  the  .10  to  > .05  level. 
a Substantive  (but  nonsignificant  at  .05  level)  effect  size  of  s .25  or  < -.25. 

1 One  of  the  treatment  group  schools  that  had  been  a K-5  school  in  prior  years  became  a K-4  school  and  continued  implementation  of  the  intervention.  The  Cohort  1 students,  who 
were  in  fifth  grade  during  the  2006-07  school  year,  went  to  an  intermediate  school  that  had  fifth  and  sixth  grades.  The  intermediate  school  did  not  implement  the  intervention.  These 
fifth-grade  students,  their  primary  caregivers,  and  their  teachers  were  included  in  the  spring  2007  data  collection  and  all  analyses  based  upon  spring  2007  data. 

2 In  Year  1 , the  statistical  model  used  to  estimate  impacts  on  the  individual  outcomes  was  re-estimated  using  a composite  of  all  the  outcome  variables  under  a domain.  The  domain 
was  found  significant  if  the  impact  on  the  composite  was  significant.  The  composite  was  formed  by  standardizing  each  outcome  variable  using  its  standard  deviation,  combining  the 
values  of  the  outcome  variables,  and  taking  the  average  of  the  final  value. 

3 Impact  on  domain  found  statistically  significant  and  negative  in  Year  3 based  on  the  fourth  heuristic  in  which  the  statistical  model  used  to  estimate  impacts  on  the  individual  outcomes 
was  re-estimated  using  a composite  of  all  the  outcome  variables  under  a domain.  The  domain  was  found  significant  if  the  impact  on  the  composite  was  significant.  The  composite  was 
formed  by  standardizing  each  outcome  variable  using  its  standard  deviation,  combining  the  values  of  the  outcome  variables,  and  taking  the  average  of  the  final  value. 

NOTE:  Abbreviations  are 
CR:  Child  Report 
PCR:  Primary  Caregiver  Report 
TRS:  Teacher  Report  on  Student 
TRCS:  Teacher  Report  on  Classroom  and  School 
ADHD:  Attention  deficit  hyperactivity  disorder 

The  +/-  signs  in  parentheses  indicate  the  direction  of  a beneficial  outcome.  All  Impact  estimates  were  calculated  using  regression  models  where  each  program  and  school  within  a 
program  was  weighted  equally.  The  standard  errors  of  all  estimates  account  for  design  effects  due  to  unequal  weighting  and  the  clustering  of  students  within  schools.  See  Table  1 .5  for 
information  about  the  measures  used  to  create  the  outcome  variables.  The  effect  size  was  calculated  by  dividing  the  estimated  Impact  by  the  standard  deviation  of  the  outcome 
measure  for  the  control  group.  The  number  of  results  found  significant  was  no  more  than  expected  by  chance. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 


Chapter  3.  Competence  Support  Program 


Chapter  3.  Competence  Support  Program 


Impacts  on  Child  Outcomes  Over  Time 

CSP  impacts  on  the  child  outcomes  over  time  were  estimated  using  growth  curve  models  by  examining 
treatment  and  control  group  differences  in  the  trajectories  of  student  outcomes  during  the  follow-up  period 
while  accounting  for  clustering  at  the  school  level.  The  growth  curve  models  are  estimated  using  a three-level 
hierarchical  linear  model,  where  Level  1 corresponds  to  time,  Level  2 to  students,  and  Level  3 to  schools 
(described  in  chapter  1). 

Table  3.19  provides  the  estimates  of  CSP  impacts  on  the  growth  in  student  outcomes  over  the  3 years.  The 
estimated  impacts  range  in  effect  size  units  (absolute  value)  from  0.00  to  0.30.  CSP  had  a statistically 
significant  beneficial  effect  on  1 of  the  18  estimated  impacts  (1  of  18  is  expected  to  be  significant  by  chance), 
Victimization  at  School  (Child  Report),  with  an  effect  size  of —0.09.  There  was  a substantively  important  but 
nonsignificant  detrimental  impact  on  Altruistic  Behavior  (Teacher  Report  on  Student). 
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Table  3.19.  Impacts  on  growth  of  child  outcomes — CSP 


Average  growth  in  the  score  per  year1 

Standard 


Scale-Report 

Mean  score  at 
implementation2 

Treatment 

Control 

Impact  on 
growth3 

Effect 

4 

size 

error  of 
impact 

p-value  of 
impact 

Social  and  Emotional  Competence  Domain 

Self-Efficacy  for  Peer  Interactions-CR  (+) 

2.94 

0.13 

0.12 

0.00 

0.00 

0.03 

0.974 

Normative  Beliefs  About  Aggression-CR  (-) 

1.20 

0.12 

0.06 

0.05 

0.11 

0.04 

0.207 

Empathy-CR  (+) 

2.40 

-0.12 

-0.12 

-0.01 

-0.03 

0.02 

0.560 

Behavior  Domain 

Altruistic  Behavior-CR  (+) 

1.41 

-0.20 

-0.16 

-0.04 

-0.04 

0.03 

0.208 

Altruistic  Behavior-PCR  (+) 

2.31 

-0.01 

-0.01 

0.01 

0.01 

0.03 

0.872 

Altruistic  Behavior-TRS  (+) 

1.40 

-0.08A 

0.07 

-0.15 

-0.30 

0.07 

0.075 

Positive  Social  Behavior-PCR  (+) 

3.01 

0.01 

0.04 

-0.02 

-0.04 

0.02 

0.247 

Positive  Social  Behavior-TRS  (+) 

2.96 

0.04 

0.01 

0.03 

0.03 

0.05 

0.580 

Problem  Behavior-CR  (-) 

0.19 

0.12 

0.08 

0.03 

0.07 

0.03 

0.273 

Problem  Behavior-PCR  (-) 

1.57 

0.01 

-0.01 

0.02 

0.05 

0.01 

0.219 

Problem  Behavior-TRS  (-) 

1.36 

0.07 

0.05 

0.02 

0.05 

0.03 

0.563 

ADHD-Related  Behavior-TRS  (-) 

1.76 

-0.02 

0.00 

-0.01 

-0.02 

0.03 

0.717 

Academics  Domain 

Engagement  with  Learning-CR  (+) 

3.69 

-0.02 

-0.01 

-0.01 

-0.01 

0.03 

0.871 

Academic  Competence  and  Motivation-TRS  (+) 

2.94 

0.01 

0.03 

-0.02 

-0.02 

0.03 

0.593 

See  notes  at  end  of  table. 
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Table  3.19.  Impacts  on  growth  of  child  outcomes — CSP — Continued 


Average  growth  in  the  score  per  year1 


Scale-Report 

Mean  score  at 
implementation2 

T reatment 

Control 

Impact  on 
growth3 

Effect 

4 

size 

Standard 
error  of 
impact 

p-value  of 
impact 

Perceptions  of  School  Climate  Domain 
Positive  School  Orientation-CR  (+) 

3.03 

-0.26 

-0.19 

-0.07 

-0.09 

0.05 

0.213 

Negative  School  Orientation-CR  (-) 

1.87 

0.08 

0.10 

-0.01 

-0.02 

0.04 

0.744 

Student  Afraid  at  School-CR  (-) 

2.41 

-0.09 

-0.03 

-0.06 

-0.06 

0.03 

0.111 

Victimization  at  School-CR  (-) 

0.82 

-0.09* 

-0.01 

-0.08 

-0.09 

0.03 

0.050 

* Treatment  group  significantly  different  from  control  group  at  the  .05  level. 

A Treatment  group  significantly  different  from  control  group  at  the  .10  to  > .05  level. 

1 Pertains  to  the  estimated  slope  of  the  outcome  for  the  treatment  or  control  groups. 

2 The  average  score  at  implementation  is  calculated  across  treatment  and  control  groups,  using  regression  models  for  adjustment  on  covariates. 

3 Estimated  difference  between  the  slope  of  the  treatment  and  control  groups. 

4 Effect  size:  The  slope  of  the  treatment  group  minus  the  slope  of  the  control  group  divided  by  the  standard  deviation  of  the  outcome  for  the  program's  control  group  (the  standard 
deviation  is  calculated  without  accounting  for  school-level  clustering  or  regression  adjustments). 

NOTE:  Abbreviations  are 
CR:  Child  Report 
PCR:  Primary  Caregiver  Report 
TRS:  Teacher  Report  on  Student 
ADHD:  Attention  deficit  hyperactivity  disorder 

The  +/-  signs  in  parentheses  indicate  the  direction  of  a beneficial  outcome.  All  impact  estimates  were  calculated  using  HLM  6.06.  Sample  weights  were  used  in  all  analyses  to  give 
(1)  each  school  equal  weight  in  each  program  (within  each  time  period)  and  (2)  each  time  period  equal  weight  within  the  analysis.  See  table  1 .5  for  information  about  the  measures 
used  to  create  the  outcome  variables. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Summary 

As  part  of  the  Social  and  Character  Development  (SACD)  Research  Program,  researchers  at  the  North 
Carolina  site  implemented  and  evaluated  CSP,  a program  focused  on  social  skills  training,  social  and 
emotional  learning,  behavior  management,  and  social  dynamics  management.  Ten  public  schools  in  two 
school  districts  in  North  Carolina  were  recruited  by  the  research  team  and  randomly  assigned  to  treatment 
and  control  conditions  to  determine  the  impact  of  CSP  on  social  and  character  development  activities  in  the 
schools  and  on  the  child  outcome  domains  of  Social  and  Emotional  Competence,  Behavior,  Academics,  and 
Perceptions  of  School  Climate. 

Analyses  of  the  initial  characteristics  of  the  sample  (students,  caregivers,  communities,  teachers,  and  schools) 
indicated  that  randomization  to  treatment  and  control  status  produced  groups  that  were  relatively  similar  at 
the  start  of  the  study  (with  2 out  of  84  comparisons  statistically  significantly  different,  fewer  than  would  be 
expected  by  chance).  The  data  on  the  initial  level  of  SACD  activity  led  to  two  findings.  First,  treatment 
teachers  reported  greater  use  of  and  training  in  SACD  activities  than  did  control  teachers,  and  they  did  so 
more  often  than  would  be  expected  by  chance  (6  out  of  62  comparisons,  with  3 expected  significant  by 
chance).  There  are  two  potential  causes  for  this  finding,  and  the  analysis  cannot  be  used  to  determine  whether 
the  reason  for  such  a difference  was  that  the  two  groups  did  differ  on  their  initial  use  of  SACD  activities  (i.e., 
that  randomization  did  not  create  similar  treatment  and  control  groups)  or  the  fact  that  the  training  of  all 
treatment  teachers  and  CSP  implementation  began  before  the  initial  data  were  collected  (by  4 weeks),  which 
may  have  influenced  the  teacher  reports.  Because  it  is  likely  (though  unproven)  that  the  training  and 
implementation  affected  the  teacher  reports,  these  data  were  not  considered  appropriate  for  use  as  a baseline 
measure  of  SACD  activities  and  training  in  the  treatment  schools. 

Second,  these  data  indicate  that  the  control  condition  for  the  SACD  project  was  not  a “no  treatment”  control 
but  a “standard  practice”  control.  Because  the  control  teachers  were  not  affected  by  the  implementation  of 
the  SACD  programs  before  data  collection,  their  reports  reflected  standard  practice  in  the  control  schools. 
Standard  practice  at  the  control  schools  included  52  percent  to  93  percent  of  teachers  reporting  the  use  of 
any  SACD  activities,  88  percent  of  teachers  reporting  the  use  of  specific  materials  in  conjunction  with  these 
activities,  100  percent  reporting  the  use  of  at  least  one  of  the  specified  instructional  strategies,  and  52  percent 
reporting  participation  in  SACD  training  over  the  past  12  months. 

Analyses  of  CSP  impacts  on  the  use  of  SACD  activities  in  the  schools  revealed  impacts  on  the  use  of  such 
activities  (1 8 out  of  90  comparisons)  and  related  materials  and  strategies  (8  out  of  87  comparisons)  across  the 
3 years,  and  use  of  more  professional  development  activities  for  treatment  teachers  (5  out  of  27 
comparisons).  These  same  measures  in  the  control  schools  across  the  3 years  of  the  study  confirm  that  use  of 
these  activities  in  the  control  schools  constituted  their  standard  practice. 

Of  the  20  childtievel  outcomes  representing  the  four  domains  of  Social  and  Emotional  Competence, 
Behavior,  Academics,  and  Perceptions  of  School  Climate  assessed  in  each  of  the  3 years  of  the  study  (a  total 
of  60  results),  CSP  had  a statistically  significant  beneficial  impact  on  Problem  Behavior  (Primary  Caregiver 
Report)  in  Year  2.  A growth  curve  analysis  was  used  to  analyze  the  change  over  time  in  these  same  outcomes 
between  initial  data  collection  and  the  final  outcome  data  collection  at  the  end  of  the  study.  Of  the  18  child- 
level  outcomes  assessed,  CSP  had  a significant  beneficial  impact  on  the  trajectory  of  1,  Victimization  at 
School  (Child  Report). 

The  SACD  evaluation  did  not  find  evidence  to  support  the  hypothesis  that  CSP  had  beneficial  impacts  on 
students’  social  and  character  development.  Such  results  could  be  caused  by  the  inability  of  the  program  to 
cause  such  change,  possibly  because  the  theory  of  action  for  the  program  is  incomplete  or  the  activities  to 
carry  out  that  theory  are  not  effective. 

However,  these  results  may  also  be  due  to  the  inability  of  the  evaluation  to  observe  such  a change  due  to  the 
control  condition,  differences  in  initial  conditions,  the  level  of  nonparticipation,  or  the  sample  size.  The 
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control  schools  continued  using  their  standard  SACD  activities,  and  these  turned  out  to  be  high  in  quantity 
and  broad  in  scope.  Under  the  Student  Citizen  Act  of  2001,  character  education  was  made  a required  part  of 
the  standard  curriculum  for  North  Carolina  schools  starting  in  the  2002-03  school  year,  and  this  may  have 
been  a factor  in  high  SACD  activity  in  the  control  schools.  While  the  CSP  program  had  significant  positive 
impact  on  the  amount  and  type  of  SACD  activities,  the  resulting  difference  in  the  amount  of  SACD  activities 
between  the  treatment  and  control  school  may  not  have  been  large  enough  to  cause  significant  differences  in 
the  student  outcomes. 

Second,  there  were  differences  between  the  treatment  and  control  groups  that  might  also  be  related  to 
differences  in  outcomes.  Treatment  teachers  had  less  teaching  experience  in  their  current  schools  and 
treatment  students  reported  greater  negative  school  orientation  (both  statistically  significant  differences, 
although  no  more  than  expected  by  chance).  For  Year  3,  the  treatment  students  at  one  school  did  not  receive 
the  intervention  when  they  were  transferred  to  a new  school  for  fifth  grade  when  their  former  school 
dropped  fifth  grade. 

Third,  about  one-third  of  the  students  in  the  sample  universe  did  not  take  part  because  of  nonconsent  or 
noncompletion  of  the  surveys.  As  a determination  could  not  be  made  as  to  whether  the  students  not  taking 
part  differed  significantly  from  those  who  did  take  part,  the  evaluation’s  results  are  valid  only  for  the  students 
who  took  part.  If  the  students  not  taking  part  were  different,  and  if  they  would  have  responded  better  to  CSP 
than  to  the  SACD  activities  occurring  in  the  control  schools,  then  the  evaluation  could  have  underestimated 
the  program’s  impact.  In  addition,  significantly  lower  percentages  of  treatment  teachers  and  students 
provided  data  in  Year  3,  and  there  were  significantly  larger  percentages  of  treatment  leavers  in  Years  2 and  3. 

Finally,  the  sample  size  of  10  schools  and  the  resulting  higher  MDES  than  for  the  multiprogram  evaluation 
may  have  reduced  the  likelihood  of  detecting  statistically  significant  effects.  However,  it  should  be  noted  that 
67  percent  of  the  MDES  for  the  60  outcomes  used  in  the  year -by-year  analysis  were  below  0.25  (42%  were 
below  0.20).  In  addition,  5 of  the  60  outcomes  were  found  to  be  substantively  important,  but  3 of  these  had 
detrimental  impacts. 
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Vanderbilt  University 
(Tennessee  Site) 


Intervention 

Researchers  at  Vanderbilt  University  (Tennessee  site)  evaluated  the  Love  In  a Big  World  (LBW)  program  as 
implemented  by  program  staff.  The  curriculum  is  based  on  research  conducted  by  the  Social  Development 
Research  Group  at  the  University  of  Washington  and  on  asset  development  research  conducted  by  the  Search 
Institute.  This  character  education  program  includes  classroom  lesson  plans,  staff  and  principal  training,  a 
peer  recognition  program,  service  projects,  assembly  programs,  motivational  morning  announcements,  and 
newsletters  that  aim  to  teach  children  positive  character  traits  and  moral  principles  and  how  to  apply  them  in 
their  lives.  Table  4.1  describes  LBW’s  general  characteristics  (panel  1),  the  types  of  instruction  and  strategies 
used  (panel  2),  the  professional  development  provided  for  those  implementing  the  program  (panel  3),  and  the 
social  and  character  development  activities  (panel  4)  and  outcomes  (panel  5)  addressed  by  the  program.  The 
program  is  designed  to  encourage  wise  decisionmaking,  improve  peer  relationships,  and  promote  dialogue 
about  character  choices.  The  program  includes  the  following  elements: 

• Adults  teach  and  model  character  traits  such  as  honesty,  kindness,  responsibility,  self-control, 
moderation,  perseverance,  respect,  fairness,  cooperation,  and  love. 

• Classroom  strategies  for  teaching  character  traits  involve  story  reading,  writing,  discussion,  song,  and 
other  classroom  activities  that  occur  daily  in  10-  to  15-minute  lessons  for  30  weeks.  Teachers  utilize 
behavior  management  strategies  to  reward  exhibition  of  character  traits  and  are  encouraged  to  model 
and  reinforce  social  skills  throughout  the  day. 

• Schoolwide  strategies  for  teaching  character  traits  include  weekly  announcements,  assemblies,  service 
projects,  parent  newsletters,  and  visual  artifacts  that  illustrate  character  traits  (e.g.,  posters  and 
banners). 

• Principals,  teachers,  and  school  staff  participate  in  a workshop  on  lesson  planning  and  on  the 
logistics  of  program  implementation  at  the  start  of  each  school  year.  In  the  first  year  of 
implementation,  this  workshop  is  3 hours  long,  and  in  subsequent  years  it  is  1 hour  long. 
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Table  4.1 . Love  In  a Big  World 

Panel  1 : General  characteristics 

Target  population 
Universal 

Program  components 

Peer:  In  and  out  of  class 

Parent  Contact  and  involvement 

Classroom:  Lessons  and  behavior  management 

Schoolwide:  Planned  events,  modeling,  and  program  artifacts 

Community:  None  or  not  major  focus 

Training:  Pretraining  and  ongoing 

Level  of  integration 

Add-on  curriculum  and  schoolwide  activities 
Flexibility 

Manualized:  Curriculum  guidebook 

Adaptability:  Less  adaptable 

See  notes  at  end  of  table. 


Panel  2:  Description  of  instruction  and  strategies 

Classroom 

Lessons 

Who  delivers:  T eacher 

Activities  and  tools:  Story  reading,  writing,  interdisciplinary  activities,  rewards,  singing 
Content:  Character  traits  and  moral  virtues 
Frequency:  Daily  10-  to  15-minute  lessons  for  30  weeks 
Strategies 

Who  delivers:  T eacher 

Activities  and  tools:  Recognition  and  reinforcement  of  good  behavior;  modeling  of  character  traits 
Frequency:  Daily 

Supplement  to  classroom 
Parent  newsletters 

Schoolwide  activities 

Weekly  announcements,  occasional  assemblies,  two  service  projects,  program  artifacts 

See  notes  at  end  of  table. 
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Table  4.1 . Love  In  a Big  World — Continued 

Panel  3:  Professional  development 

Pre-implementation 

Teachers 

Content:  Workshop  on  lesson  plans  and  logistics  of  program 
Duration:  3 hours 
Other 

Content:  Principal  and  staff  training  on  same  material  as  teachers 
Duration:  3 hours 

Ongoing  consultation 
Teachers 

Content:  Faculty  boosters  with  reminders  about  program  implementation  and  discussion  of  challenges 
Duration:  24  weekly  sessions;  biannual  meeting 
Other 

Content:  Program  coordinator;  calls  with  program  developer  to  improve  schoolwide  implementation 

Duration:  Once  every  6 months 

See  notes  at  end  of  table. 


Panel  4:  Activities  for  SACD  goals 


Violence  prevention  and  peace  promotion 

V 

Risk  prevention  and  health  promotion 

Social  and  emotional  development 

✓ 

Civic  responsibility  and  community  service 

Character  education 

V 

Behavior  management 

Tolerance  and  diversity 

See  notes  at  end  of  table. 


Panel  5:  SACD  outcomes  addressed 


Engagement  with  Learning 

Empathy 

V 

Academic  Competence  and  Motivation 

Positive  School  Orientation 

s 

Altruistic  Behavior 

✓ 

Negative  School  Orientation 

s 

Positive  Social  Behavior 

✓ 

Student  Afraid  at  School 

V 

Problem  Behavior 

✓ 

Victimization  at  School 

Self-Efficacy  for  Peer  Interactions 

Feelings  of  Safety 

V 

Normative  Beliefs  About  Aggression 

Student  Support  for  Teachers 

NOTE:  Abbreviations  are 

V : Activity  or  outcome  addressed 
Blank  cell:  Activity  or  outcome  not  addressed 
SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Sample  and  Random  Assignment 

The  Tennessee  research  team  recruited  a total  of  12  public  elementary  schools  in  2 school  districts  in  2 
separate  counties  in  Tennessee.  The  12  schools  were  randomly  assigned  to  treatment  and  control  conditions 
prior  to  the  fall  2004  data  collection  period.48  A two-step  process  was  used.  First,  a computer-generated 
pairwise  matching  algorithm  developed  by  Mathematica  Policy  Research,  Inc.  (MPR)  identified  the  best 
pairwise  matches  across  the  12  schools  based  on  (a)  school  district,  (b)  percentages  of  students  eligible  for 
free  or  reduced-price  lunch,  (c)  total  mobility  rate,  (d)  Tennessee  assessment  scores,  (e)  total  number  of 
students,  (f)  promotion  level,  and  (g)  percentages  of  students  who  were  members  of  a minority  population. 
Second,  using  the  flip  of  a coin,  1 school  in  each  matched  pair  was  assigned  to  either  the  intervention  or 
control  condition,  resulting  in  6 schools  receiving  LBW  and  6 schools  acting  as  control  schools  that 
continued  to  implement  the  social  and  character  development  activities  that  constituted  their  standard 
practice.  Assignment  to  the  treatment  or  control  condition  was  at  the  school  level  and  therefore  limited  the 
risk  of  contamination  between  treatment  and  control  classrooms.  After  the  first  year,  1 control  school 
discontinued  involvement  because  it  became  a magnet  school,  resulting  in  a total  of  1 1 schools  in  the  study, 
with  6 in  the  treatment  condition  and  5 in  the  control  condition. 

The  original  student  sample  (the  cohort  of  students  in  the  third  grade  in  the  12  schools  in  fall  2004) 
numbered  986  students  (548  treatment  and  438  control).  Table  4.2  documents  the  changes  in  the  sample  over 
the  three  spring  follow-up  data  collection  points.  Over  time,  new  entrants  to  the  cohort  became  a larger 
percentage  of  the  sample,  making  up  28  percent  of  the  sample  by  the  spring  of  Year  3.  There  were  no 
statistically  significant  differences  between  the  treatment  and  control  groups  in  the  numbers  of  new  entrants. 
The  percentage  of  the  sample  made  up  of  the  original  cohort  further  declined  as  students  left  the  schools.  By 
Year  3,  approximately  31  percent  of  the  original  sample  had  left.  In  Years  1 and  3 there  were  statistically 
significant  differences  in  “leavers”  between  treatment  and  control  groups  (in  both  cases,  there  were  fewer  in 
the  treatment  group). 


48  In  Year  2 (fall  2005),  the  Tennessee  research  team  recruited  two  more  schools,  one  assigned  to  treatment  and  one 
assigned  to  control,  which  were  followed  to  the  end  of  the  study  (spring  2007).  A description  of  this  second  cohort  and 
all  relevant  findings  can  be  found  in  appendix  A.  The  data  from  this  second  cohort  are  not  included  in  the  analyses  and 
results  reported  in  this  chapter. 
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Table  4.2.  Sample— LBW 


Year  1 Year  1 Year  2 Year  3 

(Fall  3rd  grade) (Spring  3rd  grade)  (Spring  4th  grade) (Spring  5th  grade) 


Treat-  Treat-  Treat-  Treat- 


Characteristic 

Total 

ment 

Control 

Total 

ment 

Control 

Total 

ment 

Control 

Total 

ment 

Control 

School  sample  size 

12 

6 

6 

12 

6 

6 

11 

6 

51 

11 

6 

5 

Student  sample  size 

986 

548 

438 

1,007 

565 

442 

959 

556 

403 

944 

567 

377 

Stayers 

t 

t 

t 

926 

523 

403 

758 

438 

320 

678 

403 

275 

New  entrants 

t 

t 

t 

81 

42 

39 

201 

118 

83 

266 

164 

102 

New  entrants  as  a percent  of 
spring  enrollment 

t 

t 

t 

8.0 

7.4 

8.8 

21.0 

21.2 

20.6 

28.2 

28.9 

27.1 

Total  leavers  (from  original 
cohort) 

t 

t 

t 

60 

25 

35 

228 

110 

118 

308 

145 

163 

Leavers  as  a percent  of  fall 
2004  enrollment 

t 

t 

t 

6.1 

4.6* 

8.0 

23.1 

20.1 

26.9 

31.2 

26.5** 

37.2 

Number  of  students  per  school 
(mean) 

82 

91 

73 

84 

94 

74 

87 

93 

76 

86 

95 

75 

Range  of  number  of  students  per 
school 

40-138 

57-138 

40-111 

44-141 

58-141 

44-1 1 1 

45-151 

45-151 

45-112 

34-164 

47-164 

34-113 

t Not  applicable. 

* Treatment  group  significantly  different  from  control  group  at  the  .05  level. 

**  Treatment  group  significantly  different  from  control  group  at  the  .01  level. 

1 One  control  group  school  discontinued  involvement  because  it  became  a magnet  school. 
SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Implementation 

Training 

Teachers,  principals,  and  school  staff  members  in  the  intervention  schools  received  3 hours  of  program 
implementation  training  in  the  form  of  a workshop  prior  to  the  beginning  of  the  first  school  year  (table  4.1, 
panel  3).  In  the  second  and  third  years  of  the  study,  a 1-hour  workshop  was  provided  at  the  start  of  the 
school  year.  Teachers  had  access  to  ongoing  program  implementation  support  during  the  school  year  through 
24  weekly  faculty  booster  sessions  that  were  provided  to  each  school’s  program  coordinator  and  could  be  e- 
mailed  to  the  faculty.  There  were  also  biannual  meetings  that  included  reminders  about  lesson  planning  and 
discussions  of  program  implementation  challenges.  In  addition,  the  school  program  coordinator  received 
monthly  (in  Year  1)  or  biannual  (in  later  years,  in  accordance  with  school  feedback)  calls  from  program  staff 
to  answer  questions  and  provide  assistance  or  any  required  materials. 

Data  Collection 

MPR  collected  the  multiprogram  child,  teacher,  and  school  data  at  the  Tennessee  site.  Table  4.3  shows  the 
school  year  milestones  and  dates  of  implementation  for  the  Tennessee  site.  Data  were  collected  in  the  fall  and 
spring  of  the  first  2 years  and  the  spring  of  Year  3.  The  fall  2004  multiprogram  data  collection  began  on 
August  23,  2004,  and  ended  on  September  9,  2004.  Fall  data  collection  occurred  before  implementation  of 
LBW  began.  The  spring  data  collection  window  was  from  May  2,  2005,  to  May  20,  2005.  The  program  had 
been  implemented  for  33  weeks  at  the  time  of  the  spring  data  collection  and  for  34  weeks  from  the  end  of  the 
fall  data  collection.  Year  2 followed  a similar  pattern,  with  implementation  occurring  at  the  start  of  the  school 
year,  fall  data  collection  occurring  5 weeks  later,  and  spring  data  collection  occurring  31  weeks  after  fall  data 
collection  (and  40  weeks  after  the  start  of  implementation).  In  spring  2007,  data  collection  occurred  40  weeks 
after  the  start  of  implementation.  Data  collection  took  from  3 to  5 weeks  at  each  collection  point. 
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Table  4.3.  Data  collection  dates — LBW 


Data  collection  schedule 

Year  1 Year  1 Year  2 Year  3 Year  4 

(Fall  3rd  grade)  (Spring  3rd  grade)  (Fall  4th  grade)  (Spring  4th  grade)  (Spring  4th  grade) 

School  sample  size 

12 

12 

11 

10 

10 

School  year  dates 
First  day  of  school 

7/14/04; 

7/4/041 

t 

7/26/05; 

8/11/054 

t 

7/24/06; 
8/1 0/066 

Start  of  implementation 

9/13/04 

t 

First  day 

t 

First  day 

Last  day  of  school 

t 

5/26/05; 

6/2/052 

t 

5/25/06; 

6/1/065 

5/25/07; 

6/1/077 

Data  collection 
Start 

8/23/04 

5/2/05 

8/29/05 

5/1/06 

5/1/07 

End 

9/9/04 

5/20/05 

9/23/05 

5/1 9/06 

5/1 6/07 

Calendar  weeks  from 
program  implementation 
to  start  of  fall  2004  data 
collection 

t3 

t 

t 

t 

t 

Calendar  weeks  from 
start  of  school  to  start 
of  fall  data  collection 

61 

t 

54 

t 

t 

Calendar  weeks  from  end 
of  fall  data  collection  to 
start  of  spring  data 
collection 

t 

34 

t 

31 

t 

Calendar  weeks  from 
program  implementation 
to  start  of  spring  data 
collection 

t 

33 

t 

404 

406 

t Not  applicable. 

1 Eight  schools  started  on  7/14/04  and  four  started  on  8/4/04.  The  calculation  of  weeks  from  the  start  of  school  to  the  start  of  fall 
2004  data  collection  uses  7/14/04  as  the  first  day  of  the  school  year. 

2 Eight  schools  ended  on  5/26/05  and  four  ended  on  6/2/05.  The  calculation  of  weeks  from  program  implementation  to  the  end  of  the 
school  year  uses  5/26/05  as  the  last  day  of  the  school  year. 

3 Program  implementation  occurred  after  fall  2004  data  collection. 

4 Eight  schools  started  on7/25/05  and  7/27/05  and  four  started  on  8/11/05.  The  calculation  of  weeks  from  the  start  of  school  to  the 
start  of  data  collection  uses  7/26/05  as  the  first  day  of  the  school  year. 

5 Eight  schools  ended  on  5/25/06  and  four  ended  on  6/1/06. 

6 Nine  schools  started  between  7/24/06  and  7/26/06  and  two  started  on  8/1 0/06.  The  calculation  of  weeks  from  the  start  of  school  to 
the  start  of  spring  2007  data  collection  uses  7/26/06  as  the  first  day  of  the  school  year. 

7 Six  schools  ended  on  5/25/07,  three  ended  on  5/31/07,  and  two  ended  on  6/1/07. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 


261 


Chapter  4.  Love  In  a Big  World 


Consent  Rates,  Completion  Rates,  and  Percentage  of  Sample  With  Data 

The  actual  number  of  student,  primary  caregiver,  and  teacher  reports  available  for  analysis  was  smaller  than 
the  number  in  the  sample  because  consent  and  completion  rates  were  less  than  100  percent.  Primary 
caregivers  had  to  provide  consent  before  children  could  complete  the  Child  Report,  before  teachers  could 
complete  the  Teacher  Report  on  Student,  and  before  they  themselves  completed  the  Primary  Caregiver 
Report.  Teachers  also  had  to  provide  consent  before  completing  the  Teacher  Report  on  Classroom  and 
School.  Of  those  with  consent,  not  all  completed  their  respective  reports.  Table  4.4  shows  the  consent  rates, 
completion  rates,  and  percentages  of  sample  with  data  for  each  of  the  four  reports  over  the  3 years.  For  the 
Child  Report  and  two  teacher  reports,  completion  rates  ranged  from  84  percent  to  100  percent,  so  the 
consent  rates  had  the  most  influence  on  what  percentage  of  the  sample  had  data.  There  were  no  statistically 
significant  differences  between  treatment  and  control  groups  in  consent  rates  for  any  of  these  reports.  For  the 
Primary  Caregiver  Report,  the  completion  rates  dropped  over  time  from  95  percent  to  74  percent. 

The  percentages  of  the  sample  with  Child  Report  data  ranged  from  58  percent  to  62  percent  over  the  3 years. 
The  percentages  of  students  with  information  from  the  Teacher  Report  on  Student  ranged  from  62  percent 
to  68  percent.  The  percentages  of  students  with  data  from  the  Primary  Caregiver  Report  ranged  from  46 
percent  to  65  percent  and  declined  over  time.  The  percentages  of  teachers  with  data  from  the  Teacher  Report 
on  Classroom  and  School  ranged  from  84  percent  to  97  percent.  There  were  no  statistically  significant 
differences  between  treatment  and  control  conditions  in  percentages  of  students  with  data  for  any  of  the  four 
reports. 
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Table  4.4.  Consent  rates,  completion  rates,  and  percentage  of  sample  with  data  from  each  report — LBW 


Year  1 Year  1 Year  2 Year  3 

(Fall  3rd  grade)  (Spring  3rd  grade)  (Spring  4th  grade)  (Spring  5th  grade) 

Treat-  Treat-  Treat-  Treat- 


Report 

Total 

ment 

Control 

Total 

ment 

Control 

Total 

ment 

Control 

Total 

ment 

Control 

Student  sample  size 

986 

548 

438 

1,007 

565 

442 

959 

556 

403 

944 

567 

377 

Child  Report  (percent) 

Primary  caregiver  consent  rate 

65.7 

67.9 

63.0 

66.3 

68.1 

64.0 

64.4 

65.5 

63.0 

64.9 

66.0 

63.4 

Student  completion  rate 

91.5 

90.6 

92.8 

91.6 

89.9 

94.0 

93.7 

92.3 

95.7 

90.2 

89.6 

91.2 

Students  with  data1 

60.1 

61.5 

58.4 

60.8 

61.4 

60.0 

60.4 

60.4 

60.3 

58.6 

59.1 

57.8 

Primary  Caregiver  Report  (percent) 

Primary  caregiver  consent  rate 

65.7 

67.9 

63.0 

66.2 

68.0 

64.0 

64.2 

65.1 

63.0 

63.9 

64.6 

62.9 

Primary  caregiver  completion  rate 

95.2 

95.4 

94.9 

80.2 

79.2 

81.6 

79.7 

81.2 

77.6 

73.5 

70.8 

77.6 

Primary  caregivers  with  data1 

62.6 

64.8 

59.8 

53.1 

54.0 

52.0 

51.2 

52.9 

48.9 

46.9 

45.7 

48.8 

Teacher  Report  on  Student  (percent) 

Primary  caregiver  consent  rate2 

65.7 

67.9 

63.0 

66.3 

68.1 

64.0 

64.4 

65.5 

63.0 

64.9 

66.0 

63.4 

Teacher  completion  rate 

97.8 

97.6 

98.2 

99.7 

99.5 

100.0 

100.0 

100.0 

100.0 

98.0 

96.8** 

100.0 

Students  with  data1 

64.3 

66.3 

61.9 

66.1 

67.8 

63.8 

64.4 

65.5 

63.0 

63.7 

63.8 

63.4 

Teacher  Report  on  Classroom  and  School 
(3rd-  to  5th-grade  teachers)  (percent) 

Teacher  consent  rate 

99.3 

100.0 

98.5 

100.0 

100.0 

100.0 

97.0 

97.4 

96.3 

100.0 

100.0 

100.0 

Teacher  completion  rate 

89.0 

84.0* 

95.3 

95.9 

95.2 

96.9 

90.6 

92.1 

88.5 

91.9 

90.4 

94.3 

Teachers  with  data1 

88.4 

84.0 

93.8 

95.9 

95.2 

96.9 

87.9 

89.7 

85.2 

91.9 

90.4 

94.3 

* Treatment  group  significantly  different  from  control  group  at  the  .05  level. 

**  Treatment  group  significantly  different  from  control  group  at  the  .01  level. 

1 Calculated  as  consent  rate  x completion  rate. 

2The  primary  caregiver  consent  rates  for  the  Child  Report  and  the  Teacher  Report  on  Student  are  identical,  as  the  primary  caregiver  gave  consent  to  both  together. 
SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Responses  from  students  in  the  original  cohort  (stayers)  and  new  entrants  in  the  LBW  sample  were  examined 
to  investigate  possible  differences  between  the  two  groups  in  consent  rates,  completion  rates,  and  the 
percentages  of  sample  with  data  that  might  affect  outcome  data  (table  4.5).  In  all  3 years,  stayers  had 
significantly  higher  consent  rates  (by  1 8 to  32  percentage  points)  and  higher  percentages  of  sample  with  data 
(by  8 to  33  percentage  points)  than  did  new  entrants. 
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Table  4.5.  Consent  rates,  completion  rates,  and  percentage  of  sample  with  data:  Stayers  versus  new  entrants — LBW 


Year  1 

(Spring  3rd  grade) 

Year  2 

(Spring  4th  grade) 

Year  3 

(Spring  5th  grade) 

Report 

Total 

New 

Stayers  entrants 

Total 

New 

Stayers  entrants 

Total 

New 

Stayers  entrants 

Student  sample  size 

1,007 

926 

81 

959 

758 

201 

944 

678 

266 

Child  Report  (percent) 

Primary  caregiver  consent  rate 

66.3 

68.9*** 

37.0 

64.4 

69.7*** 

44.8 

64.9 

70.1*** 

51.9 

Student  completion  rate 

91.6 

91.5 

93.3 

93.7 

93.2 

96.7 

90.2 

89.9 

91.3 

Students  with  data1 

60.8 

63.1*** 

34.6 

60.4 

64.9*** 

43.3 

58.6 

63.0*** 

47.4 

Primary  Caregiver  Report  (percent) 

Primary  caregiver  consent  rate 

66.2 

68.9*** 

35.8 

64.2 

69.4*** 

44.8 

63.9 

68.9*** 

51.1 

Primary  caregiver  completion  rate 

80.2 

80.1 

82.8 

79.7 

80.0 

77.8 

73.6 

71.5* 

80.9 

Primary  caregivers  with  data1 

53.1 

55.2*** 

29.6 

51.2 

55.5*** 

34.8 

47.0 

49.3* 

41.4 

Teacher  Report  on  Student  (percent) 

Primary  caregiver  consent  rate2 

66.3 

68.9*** 

37.0 

64.4 

69.7*** 

44.8 

64.9 

70.1*** 

51.9 

Teacher  completion  rate 

99.7 

99.8 

96.7 

100.0 

100.0 

100.0 

98.0 

98.3 

97.1 

Students  with  data1 

66.1 

68.8*** 

35.8 

64.4 

69.7*** 

44.8 

63.7 

68.9*** 

50.4 

* Stayers  significantly  different  from  new  entrants  at  the  .05  level. 

***  Stayers  significantly  different  from  new  entrants  at  the  .001  level. 

1 Calculated  as  consent  rate  x completion  rate. 

2The  primary  caregiver  consent  rates  for  the  Child  Report  and  the  Teacher  Report  on  Student  are  identical,  as  the  primary  caregiver  gave  consent  to  both  together. 
SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Fidelity  of  Implementation 

Each  year,  LBW’s  six  treatment  group  schools  were  individually  rated  for  quantity  and  quality  of  program 
implementation  by  two  raters  from  the  research  team.  The  global  measure  of  fidelity  for  the  multisite  study 
was  used;  inter-rater  reliability  was  measured  using  Cronbach’s  alpha.  In  Years  2 and  3 these  values  were  0.83 
and  0.97,  respectively;  in  Year  1,  the  inter -rater  reliability  coefficient  was  -0.08,  indicating  that  there  was  lack 
of  agreement  among  the  two  raters,  and  thus  Cronbach’s  alpha  was  not  calculated.  The  ratings  were 
combined  into  a single  consensus  rating  and  used  to  identify  schools  with  high  implementation  fidelity.  In 
Year  1 one  treatment  school  was  identified  as  having  high  fidelity  and  in  Years  2 and  3 two  treatment  schools 
were  identified  as  having  high  fidelity.  Cohen’s  kappa  was  used  as  the  measure  of  agreement  when  identifying 
schools  as  high  fidelity,  and  it  equaled  1.00  in  Years  2 and  3. 

Initial  Characteristics 

This  section  examines  the  initial  characteristics  of  the  students,  teachers,  and  schools  participating  in  the 
LBW  evaluation.  These  characteristics  were  collected  from  students  who  were  enrolled  in  the  third  grade  at 
the  study  schools  in  fall  2004,  as  well  as  from  their  primary  caregivers  and  third-grade  teachers.  In  addition, 
third-,  fourth-,  and  fifth-grade  teachers  and  principals  in  the  study  schools  provided  information  about 
activities  related  to  social  and  character  development  in  these  schools.  Documenting  the  characteristics  of 
students,  teachers,  and  schools  and  initial  measures  of  key  outcomes  at  a point  before  the  intervention  began 
operating  helps  to  determine  whether  the  random  assignment  of  schools  to  treatment  and  control  status 
produced  treatment  and  control  groups  with  similar  distributions  of  observed  characteristics.  As  noted  in  the 
following  discussion,  4 significant  differences  between  the  treatment  and  control  students,  teachers,  and 
schools  were  found  in  the  observed  characteristics,  including  the  level  of  SACD  activity  in  the  classroom  and 
school  (3  of  the  4 significant  differences  out  of  62  comparisons,  with  3 expected  to  be  significant  by  chance). 

Characteristics  of  Children,  Their  Families,  and  Communities 

There  were  no  significant  differences  between  the  treatment  and  control  groups  in  the  observed  student, 
caregiver,  and  community  characteristics  (table  4.6).  For  students,  the  mean  age  was  8.1  years.  The  sample 
contained  roughly  equal  percentages  of  girls  (52%)  and  boys  (48%).  The  sample  was  ethnically  diverse.  White 
non- Hispanic  students  made  up  65  percent  of  the  sample,  Black  non-Hispanic  students  made  up  21  percent, 
and  Hispanic  students  made  up  8 percent. 

The  sample  was  also  diverse  in  its  levels  of  family  income,  education  levels  of  primary  caregivers  of  the 
children  in  the  sample,  and  family  situation.  Twenty-eight  percent  of  children  lived  in  a household  where  the 
income  was  135  percent  of  the  federal  poverty  level  or  lower,  which  is  the  income  threshold  for  eligibility  for 
free  school  meals.  Slightly  less  than  9 percent  of  primary  caregivers  had  not  completed  high  school.  Nearly 
two-thirds  of  the  children  (64%)  lived  with  both  their  mother  and  their  father.  There  were  no  significant 
differences  between  the  treatment  and  control  groups  in  these  characteristics. 

The  mean  values  of  the  outcomes  for  children’s  behavior  and  attitudes  as  reported  by  the  primary  caregiver, 
child,  and  teacher  at  initial  data  collection  in  fall  2004  are  shown  in  table  4.7.  There  was  1 significant 
difference  (out  of  18  comparisons)  between  the  treatment  and  control  groups:  the  treatment  group  had  a 
lower  Altruistic  Behavior  score  (as  reported  by  caregivers)  than  did  the  control  group. 
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Table  4.6.  Initial  characteristics  of  children,  their  families,  and  communities — LBW 


Characteristic 

Total 

Treatment 

Control 

Student  sample  size 

617 

355 

262 

Student  demographics 

Gender  (percent) 

Male 

48.0 

47.7 

48.4 

Female 

52.0 

52.3 

51.6 

Race/ethnicity  (percent) 

White  (non-Hispanic) 

64.9 

65.7 

64.1 

Black  (non-Hispanic) 

21.1 

22.4 

19.8 

Hispanic 

7.9 

6.1 

9.8 

Other 

6.1 

5.8 

6.3 

Age  (in  years)  (mean) 

8.1 

8.1 

8.1 

Primary  caregiver  and  family  characteristics 

Primary  caregiver’s  age  (in  years)  (mean) 

35.5 

35.8 

35.2 

Primary  caregiver’s  race/ethnicity  (percent) 

White  (non-Hispanic) 

70.3 

70.0 

70.7 

Black  (non-Hispanic) 

20.6 

22.3 

19.0 

Hispanic 

6.7 

6.2 

7.2 

Other 

2.3 

1.5 

3.1 

Primary  caregiver’s  education  (percent) 

Did  not  complete  high  school 

8.8 

5.4 

12.3 

Completed  high  school  or  equivalent 

25.8 

22.3 

29.3 

Some  college 

40.4 

42.2 

38.6 

Bachelor’s  or  higher  degree 

25.0 

30.2 

19.8 

Primary  caregiver’s  employment  (percent) 

Full-time 

54.5 

56.9 

52.1 

Other 

45.5 

43.1 

47.9 

Primary  caregiver’s  marital  status  (percent) 

Married 

63.9 

65.9 

61.8 

Other 

36.1 

34.1 

38.2 

Students  who  live  in  one  household  (percent) 

91.8 

91.8 

91.8 

Number  of  individuals  in  household  (mean) 

4.3 

4.3 

4.3 

Primary  caregiver’s  relationship  to  child  (percent) 

Mother  (stepmother) 

84.9 

86.0 

83.8 

Father  (stepfather) 

10.4 

9.6 

11.2 

Other  relative/nonrelative 

4.7 

4.4 

4.9 

See  notes  at  end  of  table. 
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Table  4.6.  Initial  characteristics  of  children,  their  families,  and  communities — 
LBW— Continued 


Characteristic 

Total 

Treatment 

Control 

Student  lives  with  (percent) 

Mother  (stepmother)  and  father  (stepfather) 

64.3 

66.6 

61.9 

Mother  (stepmother)  only;  father  (stepfather)  not  present 

28.2 

27.3 

29.1 

Father  (stepfather)  only;  mother  (stepmother)  not  present 

3.8 

2.9 

4.7 

Other  relative/nonrelative,  parents  not  present 

3.7 

3.1 

4.3 

Highest  education  of  anyone  in  household  (percent) 

Did  not  complete  high  school 

5.6 

2.7 

8.4 

Completed  high  school  or  equivalent 

20.5 

18.7 

22.2 

Some  college 

40.6 

40.8 

40.5 

Bachelor’s  or  higher  degree 

33.4 

37.8 

28.9 

Total  household  income  (percent) 

Less  than  $20,000 

24.1 

19.5 

28.8 

$20,000  to  $39,999 

22.4 

20.2 

24.6 

$40,000  to  $59,999 

19.9 

19.8 

20.1 

$60,000  or  more 

33.5 

40.5 

26.5 

Income-to-poverty-threshold  ratio — Below  135  percent  (percent) 

28.1 

24.6 

31.6 

Income-to-poverty-threshold  ratio — 135  to  185  percent  (percent) 

18.1 

14.0 

22.1 

Income-to-poverty-threshold  ratio — Above  185  percent  (percent) 

53.8 

61.4 

46.3 

Alabama  Parenting  Questionnaire — Poor  Monitoring  and 

Supervision  Subscale  (mean) 

1.1 

1.1 

1.2 

Alabama  Parenting  Questionnaire — Positive  Parenting 

Subscale  (mean) 

3.5 

3.6 

3.5 

Confusion,  Hubbub,  and  Order  Scale  (mean) 

2.2 

2.2 

2.2 

Community  characteristics  (mean) 

Community  Risks  Scale 

1.2 

1.2 

1.2 

Community  Resources  Scale 

2.6 

2.6 

2.7 

Child-Centered  Social  Control  Scale 

3.1 

3.2 

3.1 

NOTE:  No  statistically  significant  differences  were  found  between  values  for  treatment  and  control  groups.  Weights,  which  assign 
equal  weight  to  each  school  within  the  program,  were  used  in  producing  the  treatment,  control,  and  overall  means.  Statistical  tests 
were  conducted  using  regressions  that  included  program  indicators  to  account  for  the  sample  design  and  adjusted  for  clustering  at 
the  school  level. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Table  4.7.  Mean  scores  and  standard  deviations  for  initial  outcome  measures  of  sample — LBW 


Outcome  measure-Report 

Range 

Total 

Mean 

SD 

Treatment 
Mean  SD 

Control 
Mean  SD 

Social  and  Emotional  Competence  Domain 

Self-Efficacy  for  Peer  Interaction-CR 

1-4 

3.0 

0.6 

3.0 

0.6 

3.0 

0.7 

Normative  Beliefs  About  Aggression-CR 

1-4 

1.2 

0.4 

1.2 

0.4 

1.2 

0.4 

Empathy-CR 

1-3 

2.5 

0.4 

2.5 

0.4 

2.5 

0.4 

Behavior  Domain 

Altruistic  Behavior-CR 

0-3 

1.4 

0.9 

1.5 

0.9 

1.4 

0.9 

Altruistic  Behavior-TRS 

1-4 

1.3 

0.4 

1.3 

0.4 

1.4 

0.4 

Altruistic  Behavior-PCR 

1-4 

2.3 

0.7 

2 2** 

0.6 

2.4 

0.7 

Positive  Social  Behavior-TRS 

1-4 

3.2 

0.7 

3.3 

0.7 

3.2 

0.7 

Positive  Social  Behavior-PCR 

1-4 

3.0 

0.5 

3.1 

0.5 

3.0 

0.5 

Problem  Behavior-CR 

0-3 

0.1 

0.3 

0.1 

0.3 

0.2 

0.3 

Problem  Behavior-TRS 

1-4 

1.3 

0.3 

1.3 

0.4 

1.3 

0.3 

Problem  Behavior-CR 

1-4 

1.6 

0.3 

1.6 

0.3 

1.6 

0.3 

ADHD-Related  Behavior-TRS 

1-4 

1.7 

0.6 

1.7 

0.7 

1.6 

0.6 

Academics  Domain 

Academic  Competence  and  Motivation-TRS 

1-5 

3.1 

0.9 

3.1 

0.9 

3.1 

0.9 

Engagement  with  Learning-CR 

1-4 

3.7 

0.5 

3.7 

0.6 

3.8 

0.5 

Perceptions  of  School  Climate  Domain 

Positive  School  Orientation-CR 

1-4 

3.3 

0.6 

3.4 

0.6 

3.3 

0.6 

Negative  School  Orientation-CR 

1-4 

1.7 

0.6 

1.7 

0.6 

1.6 

0.5 

Student  Afraid  at  School-CR 

1-4 

2.4 

0.9 

2.4 

0.9 

2.3 

0.9 

Victimization  at  School-CR 

0-3 

0.7 

0.8 

0.7 

0.8 

0.7 

0.8 

Student  sample  size — PCR 

617 

355 

262 

Student  sample  size — CR 

593 

337 

256 

Student  sample  size — TRS 

634 

363 

271 

**  Treatment  group  significantly  different  from  control  group  at  the  .01  level. 

NOTE:  Abbreviations  are 
CR:  Child  Report 
PCR:  Primary  Caregiver  Report 
TRS:  Teacher  Report  on  Student 
ADHD:  Attention  deficit  hyperactivity  disorder 
SD:  Standard  deviation 

Weights,  which  assign  equal  weight  to  each  school  within  the  program,  were  used  in  producing  the  treatment,  control,  and  overall 
means.  Statistical  tests  were  conducted  using  regressions  that  included  program  indicators  to  account  for  the  sample  design  and 
adjusted  for  clustering  at  the  school  level.  Sample  size  may  differ  for  some  outcomes  due  to  nonresponse. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Characteristics  of  Teachers  and  Schools 

Table  4.8  describes  the  third-,  fourth-,  and  fifth-grade  teachers  at  the  study  schools.  They  were  predominandy 
White  non-Hispanic  (91%)  and  female  (90%)  and  had  an  average  of  14.9  years  of  total  teaching  experience. 
More  than  half  (55%)  held  an  advanced  or  specialist  degree.  There  were  no  statistically  significant  differences 
between  the  treatment  and  control  groups  of  teachers. 

Data  regarding  school  characteristics  were  drawn  from  the  Common  Core  of  Data  in  order  to  compare 
treatment  and  control  schools.49  There  were  no  significant  differences  between  the  two  groups  of  schools  in 
terms  of  student  composition  (race/ ethnicity  and  free  lunch  eligibility),  number  of  students  enrolled,  number 
of  full-time  teachers,  Title  1 status,  or  number  of  years  the  principal  had  been  at  the  school  (see  table  4.9).  In 
addition,  there  were  no  significant  differences  between  treatment  and  control  schools  in  terms  of  location 
(urban,  suburban,  or  rural)  or  lowest  and  highest  grade  offered  (these  data  are  not  shown  in  a table). 


Table  4.8.  Initial  characteristics  of  teachers  in  sample — LBW 


Characteristic 

Total 

Treatment 

Control 

Teacher  sample  size 

129 

68 

61 

Gender  (percent) 

Male 

10.0 

8.6 

11.4 

Female 

90.0 

91.4 

88.6 

Race/ethnicity  (percent) 

White  (non-Hispanic) 

90.5 

92.7 

88.4 

Other 

9.5 

7.3 

11.6 

Number  of  years  teaching  experience  (mean) 

14.9 

14.5 

15.2 

Number  of  years  teaching  experience  in  this  school  (mean) 

10.0 

8.7 

11.2 

Type  of  teaching  certificate  (percent) 

Regular  state  certificate  or  advanced  professional  certificate 

95.5 

t 

t 

Other 

4.5 

t 

t 

Education  (percent) 

Bachelor’s  degree 

44.7 

40.0 

49.4 

Advanced  degree/other 

55.3 

60.0 

50.6 

$ Reporting  standards  not  met.  Values  suppressed  to  protect  confidentiality. 

NOTE:  No  statistically  significant  differences  were  found  between  values  for  treatment  and  control  groups.  Weights,  which  assign 
equal  weight  to  each  school  within  the  program,  were  used  in  producing  the  treatment,  control,  and  overall  means.  Statistical  tests 
were  conducted  using  regressions  that  included  program  indicators  to  account  for  the  sample  design  and  adjusted  for  clustering  at 
the  school  level.  Sample  size  may  differ  for  some  outcomes  due  to  nonresponse. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program 


49  Common  Core  of  Data  information  on  student  race/ethnicity,  percentage  eligible  for  free  or  reduced-price  lunch,  and 
number  of  full-time  teachers  were  missing  for  LBW  treatment  and  control  schools. 
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Table  4.9.  Initial  characteristics  of  schools  in  sample — LBW 


Characteristic 

Total 

Treatment 

Control 

School  sample  size 

12 

6 

6 

Student  race/ethnicity  (percent) 

White  (non-Hispanic) 

— 

— 

— 

Black  (non-Hispanic) 

— 

— 

— 

Hispanic 

— 

— 

— 

Other 

— 

— 

— 

Students  eligible  for  free  or  reduced-price  lunch  (percent) 

Number  of  students  enrolled  (mean) 

536.2 

576.5 

495.8 

Number  of  full-time  teachers  (mean) 

— 

— 

— 

Title  1 status  (percent) 

Title  1 eligible  school 

66.7 

66.7 

66.7 

Schoolwide  Title  1 

t 

0 

t 

Number  of  years  principal  has  been  at  this  school  (mean) 

6.7 

4.8 

8.5 

— Not  available. 

$ Reporting  standards  not  met.  Values  suppressed  to  protect  confidentiality. 

NOTE:  No  statistically  significant  differences  were  found  between  values  for  treatment  and  control  groups.  Weights,  which  assign 
equal  weight  to  each  school  within  the  program,  were  used  in  producing  the  treatment,  control,  and  overall  means.  Statistical  tests 
were  conducted  using  regressions  that  included  program  indicators  to  account  for  the  sample  design  and  adjusted  for  clustering  at 
the  school  level.  Sample  size  may  differ  for  some  outcomes  due  to  nonresponse. 

SOURCE:  NCES  Common  Core  of  Data  (2003-04),  the  Social  and  Character  Development  (SACD)  Research  Program. 


In  the  Teacher  Report  on  Classroom  and  School,  teachers  reported  on  nine  dimensions  of  school 
environment  (these  data  are  not  shown  in  a table):  feelings  of  safety,  adequacy  of  resources,  student  support, 
freedom  to  teach  as  desired,  affiliation  with  and  ties  to  colleagues,  innovation  regarding  new  approaches  to 
teaching,  professional  interest,  participatory  decisionmaking,  and  work  pressure.  There  were  no  statistically 
significant  differences  in  these  reports  between  treatment  and  control  schools. 

The  Level  of  SACD  in  the  Schools  Near  the  Beginning  of  the  Study 

During  the  initial  data  collection,  principals  and  teachers  reported  on  the  SACD  activities  used  in  the  schools 
and  classrooms,  the  availability  of  SACD  materials,  and  the  professional  development  provided  on  SACD. 
Table  4.10  shows  that  the  majority  of  the  school  principals  reported  activities  to  promote  six  social  and 
character  development  goals:  violence  prevention  and  peace  promotion  (92%),  social  and  emotional 
development  (92%),  character  education  (92%),  tolerance  and  diversity  (83%),  risk  prevention  and  health 
promotion  (82%),  and  civic  responsibility  and  community  service  (91%).  In  addition,  all  of  the  principals 
reported  activities  directed  toward  behavior  management.  There  were  no  statistically  significant  differences 
between  treatment  and  control  groups  in  the  percentages,  although  this  may  be  due  to  the  relatively  small 
principal  sample  size.  The  percentages  of  teachers  reporting  the  use  of  these  activities  in  their  classrooms 
ranged  from  46  percent  to  89  percent,  and  there  were  no  significant  differences  between  treatment  and 
control  teacher  reports.  With  respect  to  the  use  of  schoolwide  activities,  42  percent  to  83  percent  of  teachers 
reported  that  their  schools  used  such  activities,  and  there  were  no  significant  differences  between  treatment 
and  control  teacher  reports. 
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Table  4.10.  Principal  and  teacher  initial  reports  on  use  of  SACD  programs  or  activities  in 
sample — LBW 


SACD  program  or  activity 

Total 

Treatment 

Control 

Principal  sample  size 

12 

6 

6 

Teacher  sample  size 

129 

68 

61 

Principals  reporting  that  school  had  programs  or  activities 
to  promote  the  following  SACD  goals  (percent) 

Violence  prevention  and  peace  promotion 

91.7 

83.3 

100.0 

Social  and  emotional  development 

91.7 

83.3 

100.0 

Character  education 

91.7 

83.3 

100.0 

Tolerance  and  diversity 

83.3 

66.7 

100.0 

Risk  prevention  and  health  promotion 

81.8 

83.3 

80.0 

Civic  responsibility  and  community  service 

90.9 

83.3 

100.0 

Behavior  management 

100.0 

100.0 

100.0 

None  of  the  above 

0.0 

0.0 

0.0 

Teachers  reporting  on  using  programs  or  activities  in  their 
class  to  promote  the  following  SACD  goals  (percent) 

Violence  prevention  and  peace  promotion 

58.2 

65.3 

51.1 

Social  and  emotional  development 

55.2 

59.6 

50.7 

Character  education 

68.9 

70.3 

67.5 

Tolerance  and  diversity 

54.9 

58.8 

51.0 

Risk  prevention  and  health  promotion 

46.2 

46.9 

45.5 

Civic  responsibility  and  community  service 

55.4 

62.2 

48.6 

Behavior  management 

88.8 

92.7 

84.9 

None  of  the  above 

4.1 

t 

t 

Teachers  reporting  schoolwide  use  of  the 
following  activities  to  promote  SACD  (percent) 

Morning  announcements  or  videos 

83.1 

80.0 

86.1 

School  assemblies 

61.3 

70.5 

52.1 

School  newspapers  or  bulletins 

62.1 

61.1 

63.2 

Special  school  days 

42.0 

45.2 

38.9 

Special  events 

63.9 

71.5 

56.3 

Other  activities 

9.6 

10.3 

8.8 

$ Reporting  standards  not  met.  Values  suppressed  to  protect  confidentiality. 

NOTE:  No  statistically  significant  differences  were  found  between  values  for  treatment  and  control  groups.  Weights,  which  assign 
equal  weight  to  each  school  within  each  of  the  programs  and  to  each  program  across  programs,  were  used  in  producing  the 
treatment,  control,  and  overall  means.  Statistical  tests  were  conducted  using  regressions  that  included  program  indicators  to 
account  for  the  sample  design  and  adjusted  for  clustering  at  the  school  level.  Sample  size  may  differ  for  some  outcomes  due  to 
nonresponse. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Teachers  reported  using  a broad  range  of  teaching  materials  to  support  SACD  activities  (table  4.1 1),  including 
teacher  guides  (46%),  student  materials  (31%),  instructional  aids  (29%),  giveaways  (32%),  and  children’s 
literature  (41%).  There  was  a statistically  significant  difference  between  the  groups,  with  a greater  percentage 
of  treatment  teachers  reporting  the  use  of  children’s  literature  (one  out  of  seven  materials  asked  about)  than 
control  teachers  (58%  versus  24%). 

Teachers  also  reported  using  a wide  variety  of  teaching  strategies  (table  4.11).  Nearly  all  teachers  (99.5%) 
reported  using  any  of  the  20  strategies  asked  about,  and  teachers  used  an  average  of  11.2  strategies.  There 
were  no  significant  differences  in  the  average  number  of  strategies,  or  in  the  specific  strategies,  used  by 
treatment  versus  control  teachers. 
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Table  4.11.  Teacher  initial  reports  on  use  of  SACD  materials  and  classroom  strategies  in 
sample — LBW 


SACD  material  and  classroom  /strategy 

Total 

Treatment 

Control 

Teacher  sample  size 

129 

68 

61 

Teachers  using  the  following  materials  in  conjunction  with 
social  and  character  development  activities  (percent) 

Teacher  guides  (manuals,  curricula) 

46.4 

47.4 

45.3 

Student  materials  (workbooks,  worksheets) 

30.8 

31.9 

29.8 

Instructional  aids  (games,  software,  videos) 

29.0 

22.9 

35.1 

Giveaways  (bookmarks,  stickers) 

32.0 

30.5 

33.6 

Children’s  literature 

41.1 

58. 1** 

24.0 

Other  types  of  materials 

11.1 

8.1 

14.1 

Do  not  use  any  of  the  materials  listed  above 

22.3 

22.7 

21.9 

Teachers  using  any  of  the  strategies  listed  below  to  promote 
social  and  character  development  in  the  classroom  (percent) 

99.5 

99.0 

100.0 

Number  of  strategies  (listed  below)  used  by  teachers  to  promote 
social  and  character  development  in  the  classroom  (mean) 

11.2 

11.2 

11.1 

Teachers  using  each  of  the  following  strategies  to  promote 
social  and  character  development  (percent) 

Role-playing 

65.6 

64.1 

67.2 

Cooperative  learning 

94.7 

93.7 

95.8 

Peer  group  discussions 

88.0 

86.8 

89.3 

Direct  instruction  of  social  and  character  development 

66.8 

70.5 

63.0 

Skill  training 

29.4 

32.0 

26.7 

Incorporating  social  and  character  development  into 
academic  curriculum 

62.6 

66.7 

58.6 

Parent  training 

2.9 

t 

t 

Parent/community  involvement  in  program  development 
or  delivery 

14.3 

15.9 

12.7 

Mentoring 

46.8 

44.0 

49.5 

Good  behavior  notes  sent  home  daily  or  weekly 

80.2 

87.3 

73.1 

Presenting  role  models 

60.7 

70.2 

51.2 

Targeted  story  reading  or  writing  on  SACD  themes 

63.9 

65.8 

62.1 

Peer  mediation 

36.1 

35.1 

37.1 

Honor  roll  for  positive  behavior 

43.4 

37.1 

49.8 

Pledges  or  recitations  on  social  and  character 
development  themes 

35.3 

39.3 

31.3 

Guided  visualization 

31.1 

19.1 

43.2 

Student-led/student-assisted  instruction 

45.8 

41.6 

50.0 

Journaling 

81.5 

78.7 

84.2 

Time  out  for  negative  behavior 

95.1 

92.1 

98.1 

Daily  or  weekly  rewards  for  positive  behavior 

89.5 

92.0 

87.1 

if  Values  suppressed  to  protect  confidentiality. 

**  Treatment  group  significantly  different  from  control  group  at  the  .01  level. 

NOTE:  Weights,  which  assign  equal  weight  to  each  school  within  each  of  the  programs  and  to  each  program  across  programs,  were 
used  in  producing  the  treatment,  control,  and  overall  means.  Statistical  tests  were  conducted  using  regressions  that  included 
program  indicators  to  account  for  the  sample  design  and  adjusted  for  clustering  at  the  school  level.  Sample  size  may  differ  for  some 
outcomes  due  to  nonresponse. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Principals  and  teachers  reported  on  participation  in  and  amount  of  SACD  training  and  staff  development 
provided  over  the  previous  12  months  (table  4.12).  Principals  reported  higher  participation  rates  than  did 
teachers  (91%  versus  73%),  although  principals  and  teachers  reported  virtually  the  same  number  of  training 
hours  (5.8  versus  5.6).  There  was  a significant  difference  in  the  percentages  of  teachers  reporting  participation 
in  SACD  training,  with  more  treatment  teachers  reporting  participation  than  control  teachers  (84%  versus 
62%).  In  addition,  a significantly  greater  percentage  of  treatment  teachers  than  control  teachers  reported 
receiving  specific  training  in  one  of  the  seven  targeted  SACD  goals  (character  education:  48%  versus  18%). 


Table  4.12.  Principal  and  teacher  initial  reports  on  SACD  professional  development  in  sample — 
LBW 


SACD  professional  development 

Total 

Treatment 

Control 

Principal  sample  size 

12 

6 

6 

Teacher  sample  size 

129 

68 

61 

Principals  reporting  that  staff  participated  in  social  and  character 
development  training  within  the  past  year  (percent) 

90.9 

83.3 

100.0 

Teachers  reporting  participation  in  social  and  character 
development  training  within  the  past  12  months  (percent) 

72.9 

83.6* 

62.2 

Number  of  hours  of  social  and  character  development  training 
principals  report  were  provided  to  each  staff  person  last  year  (mean) 

5.8 

6.2 

5.4 

Number  of  hours  of  social  and  character  development  training 
teachers  report  receiving  during  the  past  12  months  (mean) 

5.6 

6.4 

4.8 

Teachers  reporting  receiving  training  in  the  past  12  months 
in  the  following  areas  (percent) 

Violence  prevention  and  peace  promotion 

21.3 

26.8 

15.8 

Social  and  emotional  development 

22.3 

21.3 

23.3 

Character  education 

33.1 

47.9** 

18.3 

Tolerance  and  diversity 

24.9 

29.4 

20.3 

Risk  prevention  and  health  promotion 

18.5 

20.0 

17.0 

Civic  responsibility  and  community  service 

7.3 

5.9 

8.6 

Behavior  management 

28.0 

33.9 

22.0 

* Treatment  group  significantly  different  from  control  group  at  the  .05  level. 
**  Treatment  group  significantly  different  from  control  group  at  the  .01  level. 


NOTE:  Weights,  which  assign  equal  weight  to  each  school  within  each  of  the  programs  and  to  each  program  across  programs,  were 
used  in  producing  the  treatment,  control,  and  overall  means.  Statistical  tests  were  conducted  using  regressions  that  included 
program  indicators  to  account  for  the  sample  design  and  adjusted  for  clustering  at  the  school  level.  Sample  size  may  differ  for  some 
outcomes  due  to  nonresponse. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 


The  data  on  the  initial  level  of  SACD  activity  emphasized  that  the  control  condition  was  a “standard  practice” 
control.  Standard  practice  at  the  control  schools  included  using  SACD  activities,  materials,  and  practices, 
along  with  professional  development  for  staff,  at  rates  and  in  types  and  amounts  similar  to  the  treatment 
schools.  For  example,  the  percentage  of  teachers  who  reported  using  programs  or  activities  to  promote 
specific  SACD  goals  ranged  from  47  percent  to  93  percent  in  the  treatment  schools  and  from  46  percent  to 
85  percent  in  the  control  schools.  The  3 significant  differences  between  the  treatment  and  control  conditions 
in  the  use  of  SACD  activities  equaled  the  number  that  would  be  expected  by  chance  (3  out  of  62 
comparisons). 
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Impacts  on  Use  of  SACD  Activities 

The  introduction  of  the  formal  LBW  program  would  be  expected  to  increase  the  use  of  SACD  activities  in 
the  treatment  schools  in  comparison  to  the  control  schools.  The  analysis  of  this  impact  is  based  on  the 
Teacher  Report  on  Classroom  and  School  (TRCS).  Every  spring,  third-,  fourth-,  and  fifth-grade  teachers 
provided  information  through  the  TRCS  about  the  social  and  character  development  activities  they  used  in 
their  classrooms.  Specifically,  information  from  the  TRCS  was  used  to  determine  the  difference  between 
treatment  and  control  teachers  in  these  areas: 

1 . the  use  of  SACD  activities  in  their  classrooms  overall  and  by  SACD  goal; 

2.  the  use  of  materials  and  strategies  to  implement  the  SACD  activities  within  classrooms  and  within 
the  entire  school; 

3.  the  use  of  staff  development  to  support  the  teachers;  and 

4.  teacher  support  for  SACD  efforts  in  the  school  and  the  use  of  practices  conducive  to  the  social  and 
character  development  of  students. 

TRCS  consent  and  completion  rates  (table  4.4)  led  to  84  percent  to  97  percent  of  all  teachers  having  data  for 
the  3 years,  with  greater  percentages  of  control  teachers  providing  data  in  Years  1 and  3.  To  estimate 
intervention  impacts  for  each  of  the  outcomes,  testing  of  the  statistical  significance  of  the  differences  in 
means  was  used.  Preliminary  analysis  indicated  little  or  no  gains  in  precision  from  using  covariates.  Before  the 
mean  differences  were  tested,  the  data  were  weighted  such  that  each  school  received  equal  weight.  Standard 
errors  of  the  impact  estimates  accounted  for  the  clustering  of  teachers  within  schools.  In  addition,  a set  of 
heuristics  (described  in  chapter  1)  was  applied  to  determine  whether  each  outcome  domain  was  statistically 
significant  after  adjustments  were  made  for  the  multiple  tests  conducted. 

Use  of  Activities 

The  percentages  of  control  teachers  who  reported  using  any  SACD  activities  in  their  classrooms  ranged  from 
82  percent  to  87  percent  over  the  3 years  (table  4.13,  panel  1).  For  the  six  individual  SACD  goals,  the  ranges 
varied  from  53  percent  to  77  percent  in  Year  1,  52  percent  to  70  percent  in  Year  2,  and  64  percent  to  81 
percent  in  Year  3.  Control  teachers’  use  of  behavior  management  activities  ranged  from  75  percent  to  84 
percent  over  this  period.  The  percentages  of  control  teachers  who  reported  using  any  SACD  activities  in  their 
classrooms  for  at  least  1 hour  per  week  (panel  2)  ranged  from  48  percent  to  80  percent  over  the  3 years.  For 
the  six  individual  SACD  goals,  the  percentages  ranged  from  6 percent  to  29  percent  in  Year  1,  14  percent  to 
25  percent  in  Year  2,  and  18  percent  to  28  percent  in  Year  3.  Control  teachers’  use  of  behavior  management 
activities  ranged  from  43  percent  to  57  percent  over  this  period.  These  findings  show  that  the  control  schools 
were  using  these  activities  as  part  of  their  standard  practice  related  to  social  and  character  development. 

For  teachers’  reported  use  of  any  SACD  activity,  48  comparisons  were  made,  with  3 expected  to  be 
significant  by  chance  (panels  1 and  2).  The  percentage  of  treatment  teachers  using  any  SACD  activity  was 
significantly  different  from  control  teachers  in  Year  2 (impact  = 17  percentage  points).  For  specific  SACD 
activities,  there  was  a significant  difference  for  character  education  in  Year  2 (impact  = 24  percentage  points) 
and  for  behavior  management  activities  in  Year  3 (impact  = 19  percentage  points),  both  favoring  treatment 
teachers.  Significant  differences  between  treatment  and  control  teachers  who  reported  use  of  these  activities 
for  at  least  1 hour  a week  occurred  in  all  years,  again  favoring  treatment  teachers.  In  Year  1,  there  was  an 
impact  on  social  and  emotional  development  (impact  = 14  percentage  points),  character  education  (impact  = 
45  percentage  points),  tolerance  and  diversity  (impact  =16  percentage  points),  and  any  activity  (impact  = 32 
percentage  points).  In  Year  2,  impacts  were  seen  for  violence  prevention  and  peace  promotion  (impact  =18 
percentage  points),  social  and  emotional  development  (impact  = 23  percentage  points),  character  education 
(impact  = 43  percentage  points),  risk  prevention  and  health  promotion  (impact  =16  percentage  points),  any 
activity  (impact  = 29  percentage  points),  and  behavior  management  (impact  = 33  percentage  points).  In  Year 
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3,  there  was  an  impact  on  behavior  management  (impact  = 26  percentage  points).  After  the  heuristics  were 
applied  to  adjust  for  multiple  comparisons,  the  data  showed  that  LBW  had  statistically  significant  impacts  in 
Years  1 and  2 on  the  domain  for  engagement  in  SACD  activities. 

Regarding  the  use  of  named  SACD  activities,  42  comparisons  were  made,  with  2 expected  to  be  significant  by 
chance  (panels  3 and  4).  Five  of  the  12  impact  estimates  in  Year  1,  6 of  the  12  in  Year  2,  and  3 of  the  12  in 
Year  3 were  statistically  significant.  An  impact  on  character  education  activities  (impact  = 44,  59,  and  31 
percentage  points)  and  any  named  activity  (impact  = 29,  45,  and  20  percentage  points)  were  seen  in  all  3 
years.  In  Year  2,  there  was  an  impact  on  risk  prevention  and  health  promotion  (impact  = 26  percentage 
points).  For  named  activities  used  at  least  1 hour  per  week,  there  were  significant  impacts  on  character 
education  (impact  = 52  percentage  points),  tolerance  and  diversity  (impact  = 21  percentage  points),  and  any 
named  activity  (impact  = 37  percentage  points)  in  Year  1;  character  education  (impact  = 43  percentage 
points)  and  any  named  activity  (impact  = 38  percentage  points)  in  Year  2;  and  character  education  (impact  = 
21  percentage  points)  in  Year  3.  LBW  had  significant  impacts  on  the  domain  for  engagement  in  named 
SACD  activities  in  Years  1 and  2. 
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Table  4.13.  Impacts  on  teacher-reported  SACD  classroom  activities — LBW 

Panel  1 : Engagement  in  any  activities  to  promote  SACD  goals1 

Year  1 Year  2 Year  3 

(Spring  3rd  grade) (Spring  4th  grade) (Spring  5th  grade) 

T reat-  T reat-  T reat- 


SACD  activity 

ment 

Control 

Impact 

p-value 

ment 

Control 

Impact 

p-value 

ment 

Control 

Impact 

p-value 

Teacher  sample  size 

85 

64 

69 

47 

75 

50 

Violence  prevention  and  peace  promotion 
(percent) 

65.6 

61.2 

4.4 

0.698 

58.8 

51.5 

7.3 

0.337 

72.0 

71.9 

0.1 

0.989 

Social  and  emotional  development 
(percent) 

66.1 

55.5 

10.6 

0.309 

66.6 

52.2 

14.4 

0.102 

64.4 

74.4 

-10.0 

0.399 

Character  education  (percent) 

92.5 

77.0 

15.4 

0.106 

93.8* 

69.6 

24.3 

0.003 

84.9 

81.4 

3.6 

0.575 

Tolerance  and  diversity  (percent) 

66.4 

62.7 

3.7 

0.704 

61.6 

59.9 

1.7 

0.862 

61.5 

70.6 

-9.2 

0.457 

Risk  prevention  and  health  promotion 
(percent) 

63.3 

53.2 

10.1 

0.119 

69.6 

52.6 

17.0 

0.114 

69.1 

64.2 

4.9 

0.692 

Civic  responsibility  and  community  service 
(percent) 

55.6 

53.1 

2.5 

0.848 

59.3 

56.0 

3.4 

0.791 

59.6 

67.5 

-7.9 

0.469 

Any  SACD  goal  (percent) 

93.5 

83.2 

10.3 

0.190 

98.3* 

81.6 

16.7 

0.024 

91.4 

86.9 

4.5 

0.457 

Behavior  management  (percent) 

85.6 

84.0 

1.7 

0.780 

90.9 

80.2 

10.7 

0.211 

93.6* 

74.7 

18.9 

0.008 

See  notes  at  end  of  table. 
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Table  4.13.  Impacts  on  teacher-reported  SACD  classroom  activities — LBW — Continued 

Panel  2:  Engagement  in  any  activities  to  promote  SACD  goals  for  at  least  1 hour  per  week 

Year  1 Year  2 Year  3 

(Spring  3rd  grade) (Spring  4th  grade) (Spring  5th  grade) 

T reat-  T reat-  T reat- 


SACD  activity 

ment 

Control 

Impact 

p-value 

ment 

Control 

Impact 

p-value 

ment 

Control 

Impact 

p-value 

Teacher  sample  size 

85 

64 

69 

47 

75 

50 

Violence  prevention  and  peace  promotion 
(percent) 

26.9 

19.9 

7.0 

0.347 

31.9* 

14.0 

17.8 

0.034 

25.2 

17.8 

7.4 

0.497 

Social  and  emotional  development 
(percent) 

32.1* 

18.5 

13.6 

0.016 

40.8* 

17.5 

23.3 

0.008 

21.3 

26.5 

-5.2 

0.565 

Character  education  (percent) 

74.1* 

28.7 

45.4 

0.000 

64.2* 

21.6 

42.5 

0.000 

37. 4A 

21.5 

15.8 

0.061 

Tolerance  and  diversity  (percent) 

32.0* 

16.1 

15.9 

0.020 

24.9 

25.4 

-0.5 

0.944 

16.9 

23.2 

-6.3 

0.512 

Risk  prevention  and  health  promotion 
(percent) 

26.1 

16.2 

9.9 

0.077 

29.9* 

14.0 

15.9 

0.046 

29.6 

28.1 

1.5 

0.920 

Civic  responsibility  and  community  service 
(percent) 

9.4 

5.7 

3.7 

0.503 

25.2 

16.8 

8.4 

0.419 

11.4 

17.9 

-6.6 

0.374 

Any  SACD  goal  (percent) 

79.9* 

47.8 

32.1 

0.001 

91.7** 

62.8 

28.9 

0.008 

84.0 

79.9 

4.1 

0.609 

Behavior  management  (percent) 

58.2 

55.4 

2.9 

0.778 

75.5** 

43.0 

32.5 

0.004 

82.2* 

56.7 

25.6 

0.025 

See  notes  at  end  of  table. 
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Table  4.13.  Impacts  on  teacher-reported  SACD  classroom  activities — LBW — Continued 

Panel  3:  Engagement  in  activities  to  promote  SACD  goals  linked  to  named  SACD  programs2 

Year  1 Year  2 Year  3 

(Spring  3rd  grade) (Spring  4th  grade) (Spring  5th  grade) 

T reat-  T reat-  T reat- 


SACD  activity 

ment 

Control 

Impact 

p-value 

ment 

Control 

Impact 

p-value 

ment 

Control 

Impact 

p-value 

Teacher  sample  size 

85 

64 

69 

47 

75 

50 

Violence  prevention  and  peace  promotion 
(percent) 

21.9 

22.4 

-0.4 

0.964 

29.1 

17.3 

11.7 

0.281 

25.9 

13.0 

12.9 

0.191 

Social  and  emotional  development 
(percent) 

25.6 

10.0 

15.6 

0.311 

31.6 

14.4 

17.2 

0.172 

10.4 

15.2 

-4.8 

0.605 

Character  education  (percent) 

62.9* 

18.5 

44.4 

0.003 

75.4* 

16.3 

59.2 

0.000 

39.2* 

8.0 

31.2 

0.047 

Tolerance  and  diversity  (percent) 

25.4 

12.0 

13.4 

0.241 

24. 6A 

9.8 

14.8 

0.060 

t 

t 

7.1 

0.361 

Risk  prevention  and  health  promotion 
(percent) 

38. 5A 

25.8 

12.7 

0.075 

52.3* 

25.9 

26.4 

0.009 

37.9 

31.2 

6.7 

0.587 

Civic  responsibility  and  community  service 
(percent) 

t 

t 

1.0 

t 

t 

t 

6.4 

0.288 

5.9 

12.3 

-6.4 

0.401 

Any  named  activity  (percent) 

69.1* 

39.7 

29.4 

0.002 

86.6* 

42.1 

44.6 

0.001 

63.0* 

43.4 

19.6 

0.021 

See  notes  at  end  of  table. 
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Table  4.13.  Impacts  on  teacher-reported  SACD  classroom  activities — LBW — Continued 


Panel  4:  Engagement  in  activities  to  promote  SACD  goals  linked  to  named  SACD  programs  for  at  least  1 hour  per  week 


Year  1 

Year  2 

Year  3 

(Spring  3rd  grade) 

(Spring  4th  grade) 

(Spring  5th  grade) 

SACD  activity 

T reat- 
ment 

Control 

Impact 

p-value 

T reat- 
ment 

Control 

Impact 

p-value 

T reat- 
ment 

Control 

Impact 

p-value 

Teacher  sample  size 

85 

64 

69 

47 

75 

50 

Violence  prevention  and  peace  promotion 
(percent) 

17. 2A 

8.4 

8.8 

0.094 

r 

t 

18.3 

0.016 

15.1 

5.1 

9.9 

0.272 

Social  and  emotional  development 
(percent) 

tA 

t 

19.6 

0.055 

23.9 

9.2 

14.6 

0.107 

7.8 

5.2 

2.6 

0.701 

Character  education  (percent) 

59.1* 

6.9 

52.3 

0.000 

53.0* 

8.2 

44.8 

0.000 

t* 

t 

21.3 

0.014 

Tolerance  and  diversity  (percent) 

22.6* 

1.7 

20.9 

0.016 

14.9 

8.8 

6.1 

0.337 

6.9 

0.0 

6.9 

t 

Risk  prevention  and  health  promotion 
(percent) 

23. 3A 

15.0 

8.3 

0.093 

22.8 

12.6 

10.2 

0.198 

19.6 

19.3 

0.3 

0.975 

Civic  responsibility  and  community  service 
(percent) 

t 

t 

1.0 

t 

4.8 

0.0 

4.8 

t 

t 

t 

-0.2 

0.909 

Any  named  activity  (percent) 

59.7* 

23.0 

36.7 

0.000 

60.3* 

22.1 

38.2 

0.000 

38.1 

20.7 

17.4 

0.131 

t Not  applicable. 

4 Reporting  standards  not  met.  Values  suppressed  to  protect  confidentiality. 

* Treatment  group  significantly  different  from  control  group  at  the  .05  level. 

**  Treatment  group  significantly  different  from  control  group  at  the  .01  level. 

A Treatment  group  significantly  different  than  control  group  at  the  .1 0 to  > .05  level. 

1 1n  Year  1 , the  omnibus  impact  for  all  the  outcomes  measured  together  was  positive  and  statistically  significant  on  the  basis  of  a multivariate  statistical  test.  In  Years  1 and  2,  at  least 
one  outcome  remained  positive  and  statistically  significant  and  no  outcome  was  negative  and  statistically  significant  after  applying  the  Benjamini-Hochberg  (1995)  procedure  to  adjust 
significance  levels  downward  to  account  for  the  multiple  testing  of  impacts. 

2 In  Years  1 and  2,  at  least  one  outcome  remained  positive  and  statistically  significant  and  no  outcome  was  negative  and  statistically  significant  after  applying  the  Benjamini-Hochberg 
(1995)  procedure  to  adjust  significance  levels  downward  to  account  for  the  multiple  testing  of  impacts. 

NOTE:  Weights,  which  assign  equal  weight  to  each  school  within  the  program,  were  used  in  producing  the  treatment,  control,  and  overall  means. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Use  of  Materials  and  Strategies 

For  use  of  materials  and  strategies  to  support  SACD  goals,  87  comparisons  were  made,  with  5 expected  to  be 
significant  by  chance.  Fifteen  significant  impacts  were  found  on  treatment  teachers’  use  of  materials  and 
strategies  in  all  3 years  (table  4.14).  In  Year  1,  more  treatment  teachers  than  control  teachers  used  teacher 
guides  (impact  = 43  percentage  points),  children’s  literature  (impact  = 38  percentage  points),  direct 
instruction  of  SACD  (impact  = 27  percentage  points),  skill  training  (impact  = 21  percentage  points),  good 
behavior  notes  (impact  = 24  percentage  points),  and  targeted  story  reading  and  writing  on  SACD  themes 
(impact  = 29  percentage  points).  On  average,  treatment  teachers  also  used  more  strategies  than  control 
teachers  (by  two  strategies).  In  Year  2,  more  treatment  teachers  than  control  teachers  continued  to  use 
children’s  literature  (impact  = 24  percentage  points),  direct  instruction  of  SACD  (impact  = 25  percentage 
points),  and  skill  training  (impact  = 22  percentage  points),  and  fewer  treatment  teachers  reported  not  using 
any  of  the  SACD  materials  (impact  = -16  percentage  points).  In  addition,  more  treatment  teachers  presented 
role  models  (impact  = 22  percentage  points).  In  Year  3,  more  treatment  teachers  used  teacher  guides  (impact 
= 22  percentage  points),  instructional  aids  (impact  =17  percentage  points),  and  direct  instruction  of  SACD 
(impact  =17  percentage  points).  LBW’s  impact  on  the  domain  of  materials  and  strategies  was  statistically 
significant  in  Year  1. 
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Table  4.14.  Impacts  on  use  of  SACD  classroom  materials  and  teaching  strategies — LBW 


Year  1 

(Spring  3rd  grade) 

Year  2 

(Spring  4th  grade) 

Year  3 

(Spring  5th  grade) 

SACD  material  and  teaching  strategy1 

Treat- 

ment 

Control 

Impact 

p-value 

T reat- 
ment 

Control  Impact 

p-value 

T reat- 
ment 

Control  Impact 

p-value 

Teacher  sample  size 

85 

64 

69 

47 

75 

50 

Use  of  SACD  materials  (percent) 

Teacher  guides  (manuals,  curricula) 

87.3* 

44.0 

43.3 

0.001 

73.5 

59.9 

13.6 

0.170 

67.9* 

46.2 

21.7 

0.028 

Student  materials  (workbooks  or  sheets) 

45.7 

54.8 

-9.2 

0.337 

46.5 

38.3 

8.2 

0.425 

46.7 

34.7 

12.0 

0.142 

Instructional  aids  (games,  software,  videos) 

43.4 

40.5 

3.0 

0.803 

40.3 

44.1 

-3.8 

0.791 

45.3* 

28.5 

16.7 

0.014 

Giveaways  (bookmarks,  stickers) 

58.2 

48.3 

9.9 

0.402 

50.2 

35.0 

15.2 

0.232 

39.7 

43.2 

-3.5 

0.750 

Children’s  literature 

75.0* 

36.9 

38.2 

0.006 

60.0* 

35.7 

24.2 

0.012 

51.3 

51.8 

-0.5 

0.953 

Other  types  of  materials 

10.1 

5.0 

5.1 

0.275 

10.2 

8.5 

1.7 

0.726 

7.7 

6.9 

0.8 

0.875 

Did  not  use  any  of  these  materials 

4.4 

20.7 

-16.2 

0.132 

6.9* 

22.4 

-15.5 

0.035 

16.8 

14.7 

2.1 

0.776 

Use  of  teaching  strategies  (percent) 

Role-playing 

83.1 

72.9 

10.2 

0.215 

70.9 

50.2 

20.7 

0.121 

74.6 

86.3 

-11.6 

0.350 

Cooperative  learning 

94.3 

92.7 

1.5 

0.669 

92.8 

90.7 

2.1 

0.680 

98.4 

95.8 

2.7 

0.405 

Peer  group  discussions 

95.5 

92.7 

2.8 

0.425 

92.1 

93.8 

-1.8 

0.796 

98.5 

97.1 

1.3 

0.658 

Direct  instruction  of  SACD 

98.1* 

71.3 

26.8 

0.003 

94.3* 

69.1 

25.2 

0.008 

95.3* 

78.7 

16.6 

0.014 

Skill  training 

57.8* 

36.4 

21.4 

0.002 

57.1* 

35.3 

21.8 

0.047 

75.2 

61.6 

13.6 

0.305 

Incorporating  SACD  into  academic 
curriculum 

77. 7A 

60.5 

17.2 

0.085 

83.1 

71.0 

12.0 

0.173 

85.2 

86.8 

-1.5 

0.816 

Parent  training 

3.9 

7.2 

-3.3 

0.458 

5.1 

12.0 

-6.9 

0.224 

22.2 

22.3 

0.0 

1.000 

Parent/community  involvement 

22.1 

25.1 

-3.1 

0.699 

18.8 

27.8 

-8.9 

0.463 

37.9 

43.5 

-5.6 

0.638 

Mentoring 

65.0 

65.6 

-0.7 

0.956 

62.6 

61.9 

0.6 

0.958 

72.5 

75.1 

-2.6 

0.834 

Good  behavior  notes  sent  home  daily 
or  weekly 

91.0* 

67.5 

23.5 

0.022 

84.3 

64.5 

19.7 

0.122 

90.7 

84.4 

6.3 

0.136 

Presenting  role  models 

84.9 

74.8 

10.1 

0.115 

80.9* 

59.4 

21.5 

0.012 

82.3 

83.6 

-1.3 

0.874 

See  notes  at  end  of  table. 
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Table  4.14.  Impacts  on  use  of  SACD  classroom  materials  and  teaching  strategies — LBW — Continued 


Year  1 Year  2 Year  3 

(Spring  3rd  grade) (Spring  4th  grade) (Spring  5th  grade) 


SACD  material  and  teaching  strategy1 

T reat- 
ment 

Control 

Impact 

p-value 

T reat- 
ment 

Control  Impact 

p-value 

T reat- 
ment 

Control  Impact 

p-value 

Use  of  teaching  strategies  (percent) — 
Continued 

Targeted  story  reading  or  writing  on  social 
and  character  development  themes 

98.0* 

69.1 

28.9 

0.001 

86.2 

76.6 

9.6 

0.301 

86.6 

90.8 

-4.2 

0.494 

Peer  mediation 

63.9 

47.6 

16.2 

0.165 

51.9 

49.5 

2.3 

0.839 

62.8 

78.3 

-15.6 

0.237 

Honor  roll  for  positive  behavior 

57.8 

57.3 

0.5 

0.955 

56.1 

59.1 

-3.0 

0.795 

67.2 

61.6 

5.6 

0.529 

Pledges  or  recitations  on  social  and 
character  development  themes 

52.7 

37.3 

15.4 

0.241 

51.7 

43.5 

8.1 

0.709 

49.0 

48.0 

1.0 

0.944 

Guided  visualization 

62.5 

52.2 

10.2 

0.191 

51.5 

42.5 

9.0 

0.477 

60.2 

57.8 

2.4 

0.776 

Student4ed/student-assisted 

instruction 

44.7 

54.6 

-9.9 

0.269 

36. 2A 

59.3 

-23.1 

0.074 

71.3 

65.8 

5.5 

0.631 

Journaling 

93. 0A 

78.8 

14.2 

0.097 

87.4 

93.6 

-6.2 

0.323 

90.3 

87.5 

2.8 

0.723 

Time  out  for  negative  behavior 

95. 7A 

86.8 

8.9 

0.086 

92.9 

94.3 

-1.5 

0.759 

98.4 

89.9 

8.5 

0.102 

Daily  or  weekly  rewards  for  positive 
behavior 

91.0 

78.0 

13.0 

0.164 

96.1 A 

83.0 

13.2 

0.098 

95.7 

94.4 

1.3 

0.793 

Any  strategy 

100.0 

100.0 

0.0 

t 

100.0 

100.0 

0.0 

t 

98.5 

100.0 

-1.5 

t 

Number  of  strategies  (mean) 

14.2* 

12.1 

2.0 

0.002 

13.5 

12.3 

1.2 

0.293 

15.0 

14.7 

0.3 

0.792 

t Not  applicable. 

* Treatment  group  significantly  different  from  control  group  at  the  .05  level. 

A Treatment  group  significantly  different  from  control  group  at  the  .10  to  > .05  level. 

1 1n  Year  1 , at  least  one  outcome  remained  positive  and  statistically  significant  and  no  outcome  was  negative  and  statistically  significant  after  applying  the  Benjamini-Hochberg  (1995) 
procedure  to  adjust  significance  levels  downward  to  account  for  the  multiple  testing  of  impacts. 

NOTE:  Weights,  which  assign  equal  weight  to  each  school  within  the  program,  were  used  in  producing  the  treatment,  control,  and  overall  means. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Regarding  the  use  of  schoolwide  strategies,  18  comparisons  were  made  between  treatment  and  control 
teacher  reports,  with  1 expected  to  be  significant  by  chance.  There  was  1 statistically  significant  difference 
between  treatment  and  control  teachers  regarding  use  of  schoolwide  strategies  in  Year  3 (these  data  are  not 
shown  in  a table),  with  treatment  teachers  reporting  more  use  of  school  assemblies  than  control  teachers 
(impact  = 22  percentage  points).  The  overall  impact  of  LBW  on  the  domain  for  use  of  schoolwide  strategies 
in  named  SACD  activities  was  significant  in  Year  2;  the  omnibus  impact  for  all  the  outcomes  measured 
together  was  positive  and  statistically  significant  on  the  basis  of  a multivariate  statistical  test. 

Participation  in  Professional  Development 

Regarding  reported  participation  in  professional  development,  27  comparisons  were  made  over  3 years,  with 
2 expected  to  be  significant  by  chance.  LBW  had  a statistically  significant  effect  on  treatment  teachers’ 
participation  in  professional  development,  with  more  treatment  teachers  reporting  SACD  training  in  the  past 
12  months  in  all  years  (impact  = 29,  31,  and  30  percentage  points)  and  more  hours  of  training  in  Years  1 and 
2 (by  5.4  and  3.3  hours,  on  average).  In  terms  of  specific  SACD  goals,  more  treatment  teachers  reported 
training  in  character  education  in  all  3 years  (impact  = 62,  34,  and  37  percentage  points).  LBW  had  a 
significant  impact  on  the  domain  of  professional  development  in  Year  1. 
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Table  4.15.  Impacts  on  teacher-reported  SACD  professional  development — LBW 


Year  1 Year  2 Year  3 

(Spring  3rd  grade) (Spring  4th  grade) (Spring  5th  grade) 

T reat-  T reat-  T reat- 


SACD  professional  development 

ment 

Control  Impact 

p-value 

ment 

Control  Impact 

p-value 

ment 

Control 

Impact  p-value 

Teacher  sample  size 

85 

64 

69 

47 

75 

50 

SACD  training  in  past  12  months  (percent) 

89.5* 

60.8 

28.7 

0.006 

68.9* 

37.7 

31.2 

0.027 

73.4* 

43.7 

30.0 

0.014 

Hours  of  SACD  training  (mean) 

8.6* 

3.1 

5.4 

0.000 

5.3* 

2.1 

3.3 

0.044 

5.6 

3.8 

1.8 

0.292 

Training  by  goal  (percent) 

Violence  prevention  and  peace  promotion 

21.9 

17.7 

4.2 

0.707 

17.0 

8.1 

8.9 

0.188 

26.2 

18.3 

7.9 

0.498 

Social  and  emotional  development 

21.8 

16.9 

4.8 

0.474 

16.5 

7.8 

8.7 

0.452 

12.7 

5.9 

6.7 

0.339 

Character  education 

77.5* 

15.3 

62.2 

0.000 

45.4* 

11.3 

34.0 

0.037 

46.7* 

9.8 

37.0 

0.015 

Tolerance  and  diversity 

31.7 

15.0 

16.8 

0.119 

11.3 

7.1 

4.3 

0.566 

14.1 

17.5 

-3.4 

0.613 

Risk  prevention  and  health  promotion 

18.3 

12.5 

5.8 

0.483 

22.1 

14.8 

7.3 

0.516 

17.5 

12.0 

5.6 

0.339 

Civic  responsibility  and  community  service 

8.2 

5.7 

2.5 

0.642 

t 

t 

2.1 

0.678 

t 

t 

-1.9 

0.623 

Behavior  management 

34.8 

22.3 

12.5 

0.184 

26.7 

18.0 

8.7 

0.227 

35.1 

20.6 

14.4 

0.344 

4 Reporting  standards  not  met.  Values  suppressed  to  protect  confidentiality. 

* Treatment  group  significantly  different  from  control  group  at  the  .05  level. 

1 1n  Year  1 , at  least  one  outcome  remained  positive  and  statistically  significant  and  no  outcome  was  negative  and  statistically  significant  after  applying  the  Benjamini-Hochberg  (1995) 
procedure  to  adjust  significance  levels  downward  to  account  for  the  multiple  testing  of  impacts. 

NOTE:  Weights,  which  assign  equal  weight  to  each  school  within  the  program,  were  used  in  producing  the  treatment,  control,  and  overall  means. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Attitudes  and  Practices 

Teachers  reported  on  their  enthusiasm  for  SACD  efforts  in  their  schools  (these  data  are  not  shown  in  a table) 
by  indicating  enthusiasm,  cooperation,  or  open  dislike.  They  also  reported  on  the  SACD  practices  of  teachers 
and  staff  members  in  their  schools  (these  data  are  not  shown  in  a table).  The  practices  included  modeling 
positive  character  and  behavior  traits  with  students  and  fellow  teachers,  involving  students  in  making 
decisions,  giving  students  a voice  in  school  governance,  the  school  encouraging  parent  involvement  in 
children’s  social  and  character  development,  and  using  developmentally  appropriate  discipline  strategies 
rather  than  punishment  for  misbehavior.  Twenty-seven  comparisons  were  made  over  3 years,  with  1 expected 
to  be  significant  by  chance.  There  were  no  statistically  significant  estimated  impacts  on  teachers’  enthusiasm 
for  SACD  efforts  in  their  schools  in  any  of  the  years,  nor  were  there  any  significant  differences  in  teacher 
reports  of  the  overall  use  of  practices  conducive  to  students’  social  and  character  development.  No  significant 
impact  was  found  on  the  domain  in  any  of  the  3 years. 

Year-by-Year  Impacts  on  Students  and  Perceptions  of  School  Climate 

The  primary  research  question  for  the  LBW  evaluation  was  this: 

What  is  the  average  effect  of  LBW  on  children’s  social  and  emotional  competence,  behavior, 
academics,  and  perceptions  of  school  climate? 

The  first  approach  to  answering  this  question  was  to  examine  the  year-by-year  impacts  of  LBW  on  these 
student  and  school  climate  outcomes  over  the  3 years  as  the  students  progressed  from  third  through  fifth 
grades. 

Equation  (2)  (described  in  chapter  1)  was  estimated  to  provide  LBW  impacts  on  the  20  outcomes  based  on 
data  from  the  11  schools  (6  treatment  and  5 control).  For  the  LBW  evaluation,  equation  (2)  excluded  the 
program  fixed  effects  (6p)  and  included  program-specific  covariates  and  random  school  effects  covariates. 
Table  4.16  lists  the  covariates  used,  with  outcomes  from  each  report  in  the  LBW  analysis. 
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Table  4.16.  Covariates  used  with  outcomes  from  each  report  for  analysis — LBW 


Potential  covariate 

CR 

outcome 

PCR 

outcome 

TRS 

outcome 

TRCS 

outcome 

Total  number 

6 

24 

21 

5 

Child-reported 

Female 

Y 

Y 

S 

Hispanic 

Y 

Y 

Y 

Black  (non-Hispanic) 

y 

y 

Y 

Other  ethnicity 

Y 

Y 

Y 

Age  in  years 

y 

Y 

Y 

Scales 

Afraid  at  School 

Altruistic  Behavior 

Y 

Empathy 

y 

Y 

Engagement  with  Learning 

Negative  School  Orientation 

y 

Normative  Beliefs  About  Aggression 

Sense  of  School  as  a Community 

Problem  Behavior 

Self-Efficacy  for  Peer  Interactions 

y 

Y 

Victimization  at  School 

y 

Primary  caregiver-reported 

Age  in  years 

y 

Completed  high  school  or  equivalent 

Some  college 

Bachelor’s  or  higher  degree 

Highest  level  of  education  in  household 

Completed  high  school  or  equivalent 

y 

Some  college 

y 

Bachelor’s  or  higher  degree 

y 

Mother  present  in  home  life 

Mother  and  father  present 

Respondent  someone  other  than  mother  or  father 

Number  of  people  in  household 

Y 

Household  income:  $20,000  to  $40,000 

y 

Y 

Household  income:  $40,000  to  $60,000 

y 

Y 

Household  income:  More  than  $60,000 

y 

Y 

Income-to-poverty-threshold  ratio:  Below  135  percent 

Y 

Income-to-poverty-threshold  ratio:  135  to  185  percent 

y 

Y 

Full-time  employment 

y 

Y 

Part-time  employment 

y 

Y 

See  notes  at  end  of  table. 
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Table  4.16.  Covariates  used  with  outcomes  from  each  report  for  analysis — LBW — Continued 


Potential  covariate 

CR 

outcome 

PCR 

outcome 

TRS 

outcome 

TRCS 

outcome 

Parental  scales 

APQ-Poor  Monitoring  and  Supervision  Subscale 

y 

Y 

APQ-Positive  Parenting  Subscale 

y 

Child-Centered  Social  Control 

Confusion,  Hubbub,  and  Order 

y 

Community  Resources 

Y 

Community  Risk 

Y 

Parent  and  Teacher  Involvement 

Y 

Child  scales 

Altruistic  Behavior 

Y 

Y 

Positive  Social  Behavior 

Problem  Behavior 

Teacher-reported 

Female 

Y 

Hispanic 

Y 

Black  (non-Hispanic) 

Y 

Other  ethnicity 

Y 

Total  teaching  experience 

Total  experience  in  current  school 

Y 

Regular  certificate 

Other  certificate 

Highest  degree-bachelor’s 

Child  scales 

Academic  Competence  and  Motivation 

ADHD-Related  Behavior 

Y 

Altruistic  Behavior 

Positive  Social  Behavior 

Y 

Problem  Behavior 

Parent  and  Teacher  Involvement 

NOTE:  Abbreviations  are 
CR:  Child  Report 
PCR:  Primary  Caregiver  Report 
TRS:  Teacher  Report  on  Student 
TRCS:  Teacher  Report  on  Classroom  and  School 
ADHD:  Attention  deficit  hypertensive  disorder 
APQ:  Alabama  Parenting  Questionnaire 
S : Covariate  used 
Blank  cell:  Covariate  not  used 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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To  assess  the  statistical  power  of  the  program -level  impact  estimates,  minimum  detectable  impacts  in  effect 
size  units  (MDES)  for  each  outcome  measure  were  calculated  for  the  LBW  evaluation  (table  4.17).  MDES 
represent  the  smallest  impacts  in  effect  size  (standard  deviation)  units  that  can  be  detected  with  a high 
probability  (80%).  The  MDES  are  primarily  a function  of  study  sample  sizes,  the  degrees  of  freedom  available 
for  statistical  tests,  and  design  effects  due  to  clustering  (Schochet  2005).  For  the  LBW  evaluation,  the  MDES 
range  from  0.099  to  0.637  for  the  child-level  outcomes  based  on  the  Child,  Caregiver,  and  Teacher  Report  on 
Student  and  from  0.536  to  0.763  for  the  school  climate  outcomes  based  on  the  Teacher  Report  on  Classroom 
and  School.  In  general,  the  MDES  for  the  school  climate  outcomes  were  larger  than  those  for  the  child-level 
outcomes. 
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Table  4.17.  Adjusted  minimum  detectable  effect  sizes  for  impact  evaluation — LBW 


Outcome  measure-Report 

Year  1 

Year  2 

Year  3 

Social  and  Emotional  Competence  Domain 

Self-Efficacy  for  Peer  Interaction-CR 

0.104 

0.107 

0.126 

Normative  Beliefs  About  Aggression-CR 

0.104 

0.178 

0.154 

Empathy-CR 

0.104 

0.154 

0.115 

Behavior  Domain 

Altruistic  Behavior-CR 

0.153 

0.228 

0.193 

Altruistic  Behavior-PCR 

0.116 

0.155 

0.148 

Altruistic  Behavior-TRS 

0.345 

0.637 

0.463 

Positive  Social  Behavior-PCR 

0.184 

0.210 

0.157 

Positive  Social  Behavior-TRS 

0.099 

0.310 

0.401 

Problem  Behavior-CR 

0.104 

0.169 

0.340 

Problem  Behavior-PCR 

0.152 

0.196 

0.135 

Problem  Behavior-TRS 

0.220 

0.104 

0.185 

ADHD-Related  Behavior-TRS 

0.227 

0.104 

0.295 

Academics  Domain 

Engagement  with  Learning-CR 

0.104 

0.107 

0.164 

Academic  Competence  and  Motivation-TRS 

0.141 

0.270 

0.191 

Perceptions  of  School  Climate  Domain 

Positive  School  Orientation-CR 

0.231 

0.202 

0.212 

Negative  School  Orientation-CR 

0.237 

0.260 

0.179 

Student  Afraid  at  School-CR 

0.223 

0.322 

0.283 

Victimization  at  School-CR 

0.297 

0.140 

0.175 

Feelings  of  Safety-TRCS 

0.763 

0.611 

0.552 

Student  Support  for  Teachers-TRCS 

0.536 

0.631 

0.726 

NOTE:  Abbreviations  are 
CR:  Child  Report 
PCR:  Primary  Caregiver  Report 
TRS:  Teacher  Report  on  Student 
TRCS:  Teacher  Report  on  Classroom  and  School 
ADHD:  Attention  deficit  hyperactivity  disorder 
The  minimum  detectable  effect  (MDE)  formula  used  in  the  calculations  is  as  follows: 


MDE  = factor{df ) ; 


Pi 


(-  + -)  + (1-Pi)(— + — ) 

Vs  r SrJ  \STnT  SrUr) 


where  sT  and  sc  are  the  number  of  treatment  and  comparison  schools;  nT  and  nc  are  the  average  number  of  students  per 
classroom;  p,is  the  intraclass  correlation  (ICC)  at  the  school  level;  and  factor(df)  is  a constant  that  depends  on  the  number  of 
degrees  of  freedom  (df)  available  for  analysis  (and  is  2.802  for  the  pooled  analysis). 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Table  4.18  provides  the  estimates  of  LBW’s  impacts  on  each  of  the  20  outcomes  over  each  of  the  3 years  (60 
impacts  in  total,  with  3 expected  to  be  statistically  significant  by  chance).  Of  the  60  results,  5 were  statistically 
significant  (2  beneficial,  3 detrimental).  In  Year  1,  LBW  had  beneficial  impacts  on  Altruistic  Behavior 
(Primary  Caregiver  Report,  effect  size  [ES]  = 0.31)  and  Student  Support  for  Teachers  (Teacher  Report  on 
Classroom  and  School,  ES  = 0.52).  In  Year  3,  LBW  had  detrimental  impacts  on  Engagement  with  Learning 
(Child  Report,  ES  = -0.35),  Positive  School  Orientation  (Child  Report,  ES  = -0.33),  and  Feelings  of  Safety 
(Teacher  Report  on  Classroom  and  School,  ES  = -0.70).  There  were  also  four  substantively  important  but 
nonsignificant  impacts  (one  beneficial,  three  detrimental):  In  Year  2,  there  was  a detrimental  impact  on 
Altruistic  Behavior  (Teacher  Report  on  Student,  ES  = -0.34)  and  a beneficial  impact  on  Student  Support  for 
Teachers  (Teacher  Report  on  Classroom  and  School,  ES  = 0.28);  in  Year  3,  there  were  detrimental  impacts 
on  Problem  Behavior  (Child  Report,  ES  = 0.31)  and  Student  Support  for  Teachers  (Teacher  Report  on 
Classroom  and  School,  ES  = -0.26).  Application  of  the  heuristic  to  adjust  for  multiple  comparisons  within 
each  outcome  domain  indicates  that  LBW  had  a significant  beneficial  impact  on  the  domain  of  Behavior  in 
Year  1 and  a significant  detrimental  impact  on  the  domain  of  Academics  in  Year  3. 
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Table  4.18.  Impacts  on  child  and  school  outcomes — LBW 


Year  1 

(Spring  3rd  grade) 

Year  2 

(Spring  4th  grade) 

Year  3 

(Spring  5th  grade) 

Scale-Report 

T reat- 
ment 

Control 

Effect 

size 

p-value 

T reat- 
ment 

Control 

Effect 

size 

p-value 

T reat- 
ment 

Control 

Effect 

size 

p-value 

Social  and  Emotional  Competence  Domain 

Self-Efficacy  for  Peer  Interactions-CR  (+) 

3.03 

3.04 

-0.01 

0.894 

3.19 

3.21 

-0.04 

0.674 

3.25 

3.24 

0.02 

0.810 

Normative  Beliefs  About  Aggression-CR  (-) 

1.24 

1.23 

0.02 

0.809 

1.26 

1.21 

0.12 

0.368 

1.35 

1.28 

0.15 

0.244 

Empathy-CR  (+) 

2.37 

2.38 

-0.02 

0.793 

2.30 

2.30 

0.00 

0.963 

2.21 

2.26 

-0.13 

0.176 

Behavior  Domain1 

Altruistic  Behavior-CR  (+) 

1.19 

1.31 

-0.15 

0.147 

1.02 

1.16 

-0.19 

0.191 

0.99 

1.13 

-0.20 

0.121 

Altruistic  Behavior-PCR  (+) 

2.33* 

' 2.13 

0.31 

0.005 

2.22 

2.11 

0.17 

0.152 

2.22 

2.23 

0.00 

0.985 

Altruistic  Behavior-TRS  (+) 

1.43 

1.41 

0.04 

0.844 

1.29 

1.46 

-0.34° 

0.270 

1.33 

1.31 

0.03 

0.914 

Positive  Social  Behavior-PCR  (+) 

3.06 

3.09 

-0.07 

0.471 

3.12 

3.04 

0.13 

0.250 

3.10 

3.05 

0.09 

0.397 

Positive  Social  Behavior-TRS  (+) 

3.17 

3.21 

-0.06 

0.343 

2.94 

3.11 

-0.24 

0.183 

3.27 

3.24 

0.06 

0.786 

Problem  Behavior-CR  (-) 

0.22 

0.21 

0.05 

0.541 

0.25 

0.23 

0.04 

0.728 

0.41 

0.29 

0.31° 

0.223 

Problem  Behavior-PCR  (-) 

1.58 

1.56 

0.06 

0.484 

1.56 

1.56 

-0.01 

0.898 

1.52 

1.57 

-0.13 

0.197 

Problem  Behavior-TRS  (-) 

1.43 

1.40 

0.08 

0.488 

1.46 

1.40 

0.14 

0.117 

1.32 

1.38 

-0.13 

0.263 

ADHD-Related  Behavior-TRS  (-) 

1.76 

1.69 

0.12 

0.263 

1.73 

1.64 

0.13 

0.105 

1.56 

1.65 

-0.16 

0.344 

Academics  Domain2 

Engagement  with  Learning-CR  (+) 

3.66 

3.66 

0.01 

0.942 

3.67 

3.69 

-0.04 

0.652 

3.54' 

‘ 3.68 

-0.35 

0.030 

Academic  Competence  and  Motivation- 

TRS  (+) 

3.06 

3.09 

-0.03 

0.666 

3.05 

2.99 

0.06 

0.622 

3.15 

3.17 

-0.02 

0.871 

See  notes  at  end  of  table. 
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Table  4.18.  Impacts  on  child  and  school  outcomes — LBW — Continued 


Year  1 

(Spring  3rd  grade) 

Year  2 

(Spring  4th  grade) 

Year  3 

(Spring  5th  grade) 

Scale-Report 

T reat- 
ment 

Control 

Effect 

size 

p-value 

T reat- 
ment 

Control 

Effect 

size 

p-value 

T reat- 

ment  Control 

Effect 

size 

p-value 

Perceptions  of  School  Climate  Domain 
Positive  School  Orientation-CR  (+) 

2.98 

2.94 

0.05 

0.692 

2.77 

2.79 

-0.03 

0.794 

2.62*  2.82 

-0.33 

0.047 

Negative  School  Orientation-CR  (-) 

1.76 

1.86 

-0.17 

0.201 

1.87 

1.89 

-0.03 

0.846 

2.05  1.95 

0.18 

0.178 

Student  Afraid  at  School-CR  (-) 

2.28 

2.30 

-0.02 

0.859 

2.28 

2.22 

0.06 

0.777 

2.26  2.16 

0.11 

0.508 

Victimization  at  School-CR  (-) 

0.73 

0.79 

-0.07 

0.657 

0.70 

0.71 

-0.01 

0.929 

0.70  0.70 

0.00 

0.993 

Feelings  of  Safety-TRCS  (+) 

3.54 

3.59 

-0.07 

0.793 

3.39 

3.45 

-0.06 

0.825 

3.32*  3.67 

-0.70 

0.046 

Student  Support  for  Teachers-TRCS  (+) 

3.72’ 

’ 3.37 

0.52 

0.022 

3.57 

3.40 

0.28° 

0.428 

3.45  3.58 

-0.26° 

0.543 

* Treatment  group  significantly  different  from  control  group  at  the  .05  level. 
a Substantive  (but  nonsignificant  at  .05  level)  effect  size  of  2 .25  or  < -.25. 

1 1n  Year  1 , at  least  one  outcome  remained  positive  and  statistically  significant  and  no  outcome  was  negative  and  statistically  significant  after  applying  the  Benjamini-Hochberg  (1995) 
procedure  to  adjust  significance  levels  downward  to  account  for  the  multiple  testing  of  impacts. 

2 Based  on  univariate  statistical  tests,  one  of  the  two  unadjusted  impacts  is  statistically  significantly  negative,  indicating  a detrimental  impact  of  the  intervention  on  child  outcomes  in 
this  domain  in  Year  3. 

NOTE:  Abbreviations  are 
CR:  Child  Report 
PCR:  Primary  Caregiver  Report 
TRS:  Teacher  Report  on  Student 
TRCS:  Teacher  Report  on  Classroom  and  School 
ADHD:  Attention  deficit  hyperactivity  disorder 

The  +/-  signs  in  parentheses  indicate  the  direction  of  a beneficial  outcome.  All  impact  estimates  were  calculated  using  regression  models  where  each  program  and  school  within  a 
program  was  weighted  equally.  The  standard  errors  of  all  estimates  account  for  design  effects  due  to  unequal  weighting  and  the  clustering  of  students  within  schools.  See  table  1 .5  for 
information  about  the  measures  used  to  create  the  outcome  variables.  The  effect  size  was  calculated  by  dividing  the  estimated  impact  by  the  standard  deviation  of  the  outcome 
measure  for  the  control  group.  The  number  of  results  found  significant  was  no  more  than  expected  by  chance. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Impacts  on  Child  Outcomes  Over  Time 

LBW’s  impacts  on  the  child  outcomes  over  time  were  estimated  using  growth  curve  models  by  examining 
treatment  and  control  group  differences  in  the  trajectories  of  student  outcomes  during  the  follow-up  period 
while  accounting  for  clustering  at  the  school  level.  The  growth  curve  models  are  estimated  using  a three-level 
hierarchical  linear  model,  where  Level  1 corresponds  to  time,  Level  2 to  students,  and  Level  3 to  schools 
(described  in  chapter  1). 

Table  4.19  provides  the  estimates  of  LBW  impacts  on  the  growth  in  student  outcomes  over  the  3 years.  The 
estimated  impacts  range  in  effect  size  units  (absolute  value)  from  0.01  to  0.13.  One  of  the  18  estimated  LBW 
intervention  impacts  on  the  trajectories  of  child  outcomes  was  statistically  significant  (no  more  than  expected 
by  chance);  the  program  had  a detrimental  impact  on  Positive  School  Orientation  (Child  Report,  ES  = -0.13). 
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Table  4.19.  Impacts  on  growth  of  child  outcomes — LBW 


Scale-Report 

Average  growth  in  the  score  per  year1 

Mean  score  at 
implementation2 

T reatment 

Control 

Impact  on 
growth3 

Effect 

4 

size 

Standard 
error  of 
impact 

p-value  of 
impact 

Social  and  Emotional  Competence  Domain 

Self-Efficacy  for  Peer  Interactions-CR  (+) 

2.96 

0.08 

0.12 

-0.04 

-0.04 

0.03 

0.298 

Normative  Beliefs  About  Aggression-CR  (-) 

1.17 

0.06 

0.04 

0.01 

0.03 

0.03 

0.620 

Empathy-CR  (+) 

2.48 

-0.10 

-0.10 

-0.01 

-0.02 

0.02 

0.693 

Behavior  Domain 

Altruistic  Behavior-CR  (+) 

1.39 

-0.20 

-0.16 

-0.04 

-0.04 

0.07 

0.591 

Altruistic  Behavior-PCR  (+) 

2.29 

-0.08 

-0.07 

-0.01 

-0.01 

0.04 

0.861 

Altruistic  Behavior-TRS  (+) 

1.40 

0.00 

-0.01 

0.01 

0.02 

0.07 

0.914 

Positive  Social  Behavior-PCR  (+) 

3.07 

0.04 

0.02 

0.03 

0.04 

0.04 

0.486 

Positive  Social  Behavior-TRS  (+) 

3.12 

0.02 

-0.01 

0.04 

0.04 

0.09 

0.691 

Problem  Behavior-CR  (-) 

0.14 

0.06 

0.08 

-0.02 

-0.04 

0.04 

0.706 

Problem  Behavior-PCR  (-) 

1.59 

-0.03 

-0.02 

-0.01 

-0.01 

0.02 

0.813 

Problem  Behavior-TRS  (-) 

1.36 

0.02 

0.04 

-0.03 

-0.06 

0.02 

0.242 

ADHD-Related  Behavior-TRS  (-) 

1.71 

-0.09 

-0.01 

-0.08 

-0.11 

0.05 

0.144 

Academics  Domain 

Engagement  with  Learning-CR  (+) 

3.75 

-0.08 

-0.04 

-0.04 

-0.06 

0.03 

0.247 

Academic  Competence  and  Motivation-TRS  (+) 

3.11 

0.03 

0.08 

-0.05 

-0.05 

0.04 

0.263 

See  notes  at  end  of  table. 
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Table  4.19.  Impacts  on  growth  of  child  outcomes — LBW — Continued 


Average  growth  in  the  score  per  year1 


Scale-Report 

Mean  score  at 
implementation2 

T reatment 

Control 

Impact  on 
growth3 

Effect 

4 

size 

Standard 
error  of 
impact 

p-value  of 
impact 

Perceptions  of  School  Climate  Domain 
Positive  School  Orientation-CR  (+) 

3.23 

-0.31* 

-0.20 

-0.11 

-0.13 

0.04 

0.016 

Negative  School  Orientation-CR  (-) 

1.68 

0.1 7A 

0.11 

0.06 

0.08 

0.03 

0.090 

Student  Afraid  at  School-CR  (-) 

2.35 

-0.03 

-0.07 

0.04 

0.03 

0.03 

0.232 

Victimization  at  School-CR  (-) 

0.68 

0.08A 

-0.03 

0.11 

0.10 

0.05 

0.061 

* Treatment  group  significantly  different  from  control  group  at  the  .05  level. 

A Treatment  group  significantly  different  from  control  group  at  the  .10  to  > .05  level. 

1 Pertains  to  the  estimated  slope  of  the  outcome  for  the  treatment  or  control  groups. 

2 The  average  score  at  implementation  is  calculated  across  treatment  and  control  groups,  using  regression  models  for  adjustment  on  covariates. 

3 Estimated  difference  between  the  slope  of  the  treatment  and  control  groups. 

4 Effect  size:  The  slope  of  the  treatment  group  minus  the  slope  of  the  control  group  divided  by  the  standard  deviation  of  the  outcome  for  the  program's  control  group  (the  standard 
deviation  is  calculated  without  accounting  for  school-level  clustering  or  regression  adjustments). 

NOTE:  Abbreviations  are 
CR:  Child  Report 
PCR:  Primary  Caregiver  Report 
TRS:  Teacher  Report  on  Student 
ADHD:  Attention  deficit  hyperactivity  disorder 

The  +/-  signs  in  parentheses  indicate  the  direction  of  a beneficial  outcome.  All  impact  estimates  were  calculated  using  HLM  6.06.  Sample  weights  were  used  in  all  analyses  to  give  (1) 
each  school  equal  weight  in  each  program  (within  each  time  period)  and  (2)  each  time  period  equal  weight  within  the  analysis.  See  table  1 .5  for  information  about  the  measures  used 
to  create  the  outcome  variables. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Summary 

As  part  of  the  Social  and  Character  Development  (SACD)  Research  Program,  researchers  at  the  Tennessee 
site  implemented  and  evaluated  the  LBW  program.  This  program  focused  on  character  education  by  teaching 
children  positive  character  traits  and  moral  virtues  and  how  to  apply  them  in  their  day-to-day  activities. 
Twelve  public  schools  in  two  school  districts  in  two  separate  counties  in  Tennessee  were  recruited  by  the 
Tennessee  research  team  and  randomly  assigned  to  treatment  and  control  conditions  to  determine  the  LBW 
impact  on  social  and  character  development  activities  in  the  schools  and  on  the  child  outcome  domains  of 
Social  and  Emotional  Competence,  Behavior,  Academics,  and  Perceptions  of  School  Climate. 

Analyses  of  the  initial  characteristics  of  the  sample  (students,  caregivers,  communities,  teachers,  and  schools) 
indicated  that  randomization  to  treatment  and  control  status  produced  groups  that  were  relatively  similar  on 
the  observed  characteristics  at  the  start  of  the  study  (1  statistically  significant  difference  in  children’s  behavior 
was  found  out  of  83  comparisons,  fewer  than  the  4 that  would  be  expected  by  chance).  Documentation  of 
the  initial  level  of  social  and  character  development  activities  in  the  schools  also  revealed  few  differences  (3, 
no  more  than  expected  by  chance  out  of  62  comparisons)  between  the  treatment  and  control  teachers  and 
classrooms.  This  is  important  for  two  reasons:  one,  it  indicates  that  randomization  created  comparable  groups 
near  the  start  of  the  study,  and  two,  it  shows  that  treatment  and  control  schools  both  had  high  levels  of 
SACD  activities  near  the  start  of  the  study,  indicating  that  social  and  character  development  activities  are  part 
of  the  “standard  practice”  of  these  schools  in  Tennessee.  Standard  practice  at  all  of  the  schools  included 
reports  that  46  percent  to  89  percent  of  teachers  used  SACD  activities,  78  percent  of  teachers  used  specific 
materials  in  conjunction  with  these  activities,  96  percent  used  at  least  one  of  the  specified  instructional 
strategies,  and  73  percent  participated  in  SACD  training  over  the  past  12  months. 

Analyses  of  LBW  impacts  on  use  of  SACD  activities  in  the  schools  revealed  impacts  on  the  use  of  such 
activities  (27  out  of  90)  and  related  materials  and  strategies  (14  out  of  87)  across  the  3 years,  and  use  of  more 
professional  development  activities  for  treatment  teachers  than  control  teachers  in  all  years  (8  out  of  27). 
These  same  measures  in  the  control  schools  across  the  3 years  of  the  study  confirmed  that  use  of  these 
activities  in  the  control  schools  constituted  their  standard  practice. 

Of  the  20  child-level  outcomes  representing  the  four  domains  of  Social  and  Emotional  Competence, 
Behavior,  Academics,  and  Perceptions  of  School  Climate  assessed  in  each  of  the  3 years  of  the  study  (a  total 
of  60  results),  5 were  statistically  significant  (2  beneficial  and  3 detrimental).  A growth  curve  analysis  was  used 
to  analyze  the  change  over  time  in  these  same  outcomes  between  initial  data  collection  and  the  final  outcome 
data  collection  at  the  end  of  the  study.  One  of  the  18  child-level  outcomes  assessed  showed  a significant 
detrimental  impact  of  the  LBW  program. 

The  SACD  evaluation  did  not  find  evidence  to  support  the  hypothesis  that  LBW  had  beneficial  impacts  on 
students’  social  and  character  development.  Such  results  could  be  caused  by  the  inability  of  the  program  to 
cause  such  change,  possibly  because  the  theory  of  action  for  the  program  is  incomplete  or  the  activities  to 
carry  out  that  theory  are  not  effective. 

However,  these  results  may  also  be  due  to  the  inability  of  the  evaluation  to  observe  such  a change  due  to  the 
control  condition,  the  level  of  nonparticipation,  or  the  sample  size.  The  control  schools  continued  using  their 
standard  SACD  activities,  and  these  turned  out  to  be  high  in  quantity  and  broad  in  scope.  While  LBW  had  a 
significant  positive  impact  on  the  amounts  and  types  of  SACD  activities,  the  resulting  difference  in  the 
amount  of  SACD  activities  between  the  treatment  and  control  schools  may  not  have  been  large  enough  to 
cause  significant  differences  in  the  student  outcomes.  In  addition,  about  one-third  of  the  students  in  the 
sample  universe  did  not  take  part  because  of  nonconsent  or  noncompletion  of  the  surveys.  As  a 
determination  could  not  be  made  as  to  whether  the  students  not  taking  part  significantly  differed  from  those 
who  did  take  part,  the  evaluation’s  results  are  valid  only  for  the  students  who  took  part.  If  the  students  not 
taking  part  were  different,  and  if  they  would  have  responded  better  to  LBW  than  to  the  SACD  activities 
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occurring  in  the  control  schools,  then  the  evaluation  could  have  underestimated  the  program’s  impact.  Third, 
the  sample  size  of  12  schools  and  the  resulting  higher  MDES  compared  to  those  for  the  multiprogram 
evaluation  may  have  reduced  the  likelihood  of  detecting  statistically  significant  effects.  However,  it  should  be 
noted  that  70  percent  of  the  MDES  for  the  60  outcomes  used  in  the  year-by-year  analysis  were  below  0.25 
(53%  were  below  0.20).  In  addition,  none  of  the  60  outcomes  were  found  to  be  substantively  important. 
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University  of  Illinois  at  Chicago/Oregon  State  University 

(Illinois  Site) 


Intervention 

Researchers  at  the  University  of  Illinois  at  Chicago/Oregon  State  University  (Illinois  site)  evaluated  the 
Positive  Action  (PA)  program  (Flay,  Allred,  and  Ordway  2001).  The  PA  program  consists  of  a school  and  home 
curriculum  and  a school  climate  program  for  teaching  children  about  the  benefits  of  physical,  intellectual, 
social,  and  emotional  positive  actions,  and  for  creating  a supportive  learning  environment  at  school  and  at 
home.  The  curriculum  is  designed  to  build  children’s  self-concept  and  self-management  through  identifying 
positive  thoughts,  feelings,  and  actions.  Table  5.1  describes  the  PA  program’s  general  characteristics  (panel  1), 
the  types  of  instruction  and  strategies  used  (panel  2),  the  professional  development  provided  for  those 
implementing  the  program  (panel  3),  and  the  social  and  character  development  activities  (panel  4)  and 
outcomes  (panel  5)  addressed  by  the  program. 

• Teachers  conduct  15-  to  20-minute  lessons,  4 days  per  week,  with  students.  Lessons  focus  on 
positive  actions  for  a healthy  body  and  mind,  social/ emotional  actions  for  managing  oneself 
responsibly,  ways  to  get  along  with  others,  how  to  be  honest  with  oneself  and  others,  and  setting 
goals  for  self-improvement. 

• Teachers  engage  in  behavior  management  practices  by  recognizing  and  reinforcing  positive  behavior 
and  by  modeling  and  reinforcing  social  skills  throughout  the  day. 

• The  school  climate  program  is  designed  to  reinforce  positive  actions  throughout  the  school,  using 
activities  such  as  peer  tutoring  and  mentoring,  assemblies,  service  projects,  PA  days,  and  visual 
artifacts  that  illustrate  positive  actions. 

• The  parent  and  community  program  includes  parent  newsletters  and  parent  night,  which  are 
designed  to  carry  classroom  concepts  into  homes  and  to  help  families  learn  and  practice  positive 
actions  in  their  communities. 

Teachers  and  staff  receive  a 4-hour  training  on  the  methods  of  instruction,  role  modeling,  use  of  positive 
actions,  and  use  of  behavior  management  strategies  at  the  beginning  of  the  school  year  and  a 2-hour  training 
(refresher  for  teachers  who  continue  in  the  same  schools,  new  for  new  teachers)  at  the  beginning  of  each 
subsequent  year.  Principals  and  the  PA  coordinator  for  each  school  receive  an  additional  2 hours  of  training 
on  the  program  adoption  process  in  a multischool  setting.  School  counselors  receive  a kit  that  includes  the 
curriculum  and  suggestions  for  activities  with  children  and  family  groups.  Ongoing  consultation  is  provided 
to  school  staff  by  the  PA  research  team  as  needed,  as  often  as  once  per  month. 
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Table  5.1 . Positive  Action 

Panel  1 : General  characteristics 

Target  population 
Universal 

Program  components 

Peer:  In  and  out  of  class 

Parent'.  Training,  contact,  and  involvement 

Classroom:  Lessons  and  behavior  management 

Schoolwide:  Planned  events,  includes  program  coordinator  and  committee 
Community:  Committees  and  groups 
Training:  Pretraining  and  ongoing 

Level  of  integration 

Add-on  curriculum  and  schoolwide  activities 
Flexibility 

Manualized:  Scripted  curriculum  and  guidebook 

Adaptability:  Program  staff  and  principal  adapt 

See  notes  at  end  of  table. 


Panel  2:  Description  of  instruction  and  strategies 

Classroom 

Lessons 

Who  delivers:  T eacher 

Activities  and  tools:  Direct  instruction,  story  reading,  writing,  role-playing,  discussion,  singing,  games, 
worksheets,  puppets,  plays,  poems 

Content:  Self-concept,  physical  health,  intellectual  growth,  emotional  and  behavioral  self-regulation, 
getting  along  with  others  (social  skills),  self-honesty,  self-improvement,  goal  setting,  character  traits 

Frequency:  1 5-  to  20-minute  lessons,  4 days  per  week 
Strategies 

Who  delivers:  T eacher 

Activities  and  tools:  Teaches,  practices,  recognizes,  and  models  positive  actions  with  curriculum  and 
climate  activities  and  materials 

Frequency:  Daily 


Supplement  to  classroom 

Parent  newsletters,  parent  manual  with  PA  activities,  letters,  strategies,  and  parent  night 


Schoolwide  activities 

Occasional  assemblies,  service  projects,  PA  days  and  year-end  event,  principal  climate  program  with 

reinforcement  recognition  activities 

See  notes  at  end  of  table. 
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Table  5.1 . Positive  Action — Continued 

Panel  3:  Professional  development 

Pre-implementation 

Teachers 

Content  Training  on  concepts  and  delivery  of  curriculum,  school  climate  activities  to  reinforce  positive 
behaviors  and  parent  involvement 
Duration:  Vz  day 
Other 

Content:  Principal  training  and  appointing  committees  on  coordinating  school  climate  activities  with 
family  groups 

Duration:  Vz  day 

Ongoing  consultation 
Teachers 

Content:  Group  session  to  provide  technical  assistance  and  share  experiences  and  challenges;  visits 
for  technical  support 

Duration:  Up  to  Vz  day  once  per  year;  up  to  Vz  day  visits  to  schools  monthly 
Other 

Content:  Meeting  of  principals  and  coordinators  to  provide  technical  assistance  and  share  successes 
and  challenges 

Duration:  1 session  of  3 hours  per  year 

See  notes  at  end  of  table. 


Panel  4:  Activities  for  SACD  goals 


Violence  prevention  and  peace  promotion 

✓ 

Risk  prevention  and  health  promotion 

✓ 

Social  and  emotional  development 

✓ 

Civic  responsibility  and  community  service 

✓ 

Character  education 

✓ 

Behavior  management 

✓ 

Tolerance  and  diversity 

✓ 

See  notes  at  end  of  table. 


Panel  5:  SACD  outcomes  addressed 


Engagement  with  Learning 

✓ 

Empathy 

✓ 

Academic  Competence  and  Motivation 

✓ 

Positive  School  Orientation 

✓ 

Altruistic  Behavior 

✓ 

Negative  School  Orientation 

✓ 

Positive  Social  Behavior 

✓ 

Student  Afraid  at  School 

✓ 

Problem  Behavior 

✓ 

Victimization  at  School 

✓ 

Self-Efficacy  for  Peer  Interactions 

✓ 

Feelings  of  Safety 

✓ 

Normative  Beliefs  About  Aggression 

✓ 

Student  Support  for  Teachers 

NOTE:  Abbreviations  are 
PA:  Positive  Action 
v : Activity  or  outcome  addressed 
Blank  cell:  Activity  or  outcome  not  addressed 
SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Sample  and  Random  Assignment 

The  Illinois  research  team  recruited  a total  of  14  public  elementary  schools  in  a single  large,  urban  school 
district  in  Illinois.  The  researchers  deliberately  selected  a sample  of  high-risk  schools  from  Chicago  that  had 
low  achievement  scores  and  families  with  low  household  incomes.  The  14  schools  were  randomly  assigned  to 
treatment  and  control  conditions  prior  to  the  fall  2004  data  collection.  A two-step  process  was  used.  First,  a 
computer-generated  pairwise  matching  algorithm  developed  by  Mathematica  Policy  Research,  Inc.  (MPR)  was 
used  to  identify  the  best  pairwise  matches  across  the  14  schools  based  on  variables  identified  by  the  Illinois 
research  team.  The  variables  used  in  the  pairwise  matching  for  Illinois  included  these:  (a)  school  schedule, 
(b)  percentage  of  White  students,  (c)  percentage  of  Black  (non-Hispanic)  students,  (d)  percentage  of  Hispanic 
students,  (e)  percentage  of  Asian  students,  (f)  achievement  scores,  (g)  percentage  of  attendance, 
(h)  percentage  of  truancy,  (i)  percentage  of  poverty,  (j)  percentage  of  mobility,  (k)  total  enrollment, 
0 percentage  of  parent  participation,  (m)  quality  of  teachers,  and  (n)  rate  of  crime  in  the  community.  Second, 
using  a random  number  generator,  1 school  in  each  matched  pair  was  assigned  to  either  the  intervention  or 
control  condition.  Seven  schools  received  the  PA  program  and  7 schools  acted  as  control  schools  and 
continued  to  implement  the  social  and  character  development  activities  that  constituted  their  standard 
practice.  Assignment  to  treatment  or  control  condition  was  at  the  school  level  and  therefore  limited  the  risk 
of  contamination  between  treatment  and  control  classrooms. 

The  original  student  sample  (the  cohort  of  students  in  the  third  grade  in  the  14  schools  in  fall  2004) 
numbered  811  students  (410  treatment  and  401  control).  Table  5.2  documents  the  change  in  the  sample  over 
the  three  spring  follow-up  data  collection  periods.  Over  time,  new  entrants  to  the  cohort  became  a larger 
percentage  of  the  sample,  eventually  making  up  39  percent  of  the  sample  by  the  spring  of  Year  3.  There  were 
no  statistically  significant  differences  between  the  treatment  and  control  groups  in  the  number  of  new 
entrants.  The  percentage  of  the  sample  made  up  of  the  original  cohort  further  declined  as  students  left  the 
schools.  By  Year  3,  approximately  50  percent  of  the  original  sample  had  left.  In  Year  2 there  was  a statistically 
significant  difference  between  treatment  and  control  groups  in  “leavers,”  with  fewer  in  the  treatment  group. 
This  difference  had  disappeared  by  the  end  of  Year  3. 
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Table  5.2.  Sample — PA 


Year  1 Year  1 Year  2 Year  3 

(Fall  3rd  grade) (Spring  3rd  grade)  (Spring  4th  grade)  (Spring  5th  grade) 


Treat-  Treat-  Treat-  Treat- 


Characteristic 

Total 

ment 

Control 

Total 

ment 

Control 

Total 

ment 

Control 

Total 

ment 

Control 

School  sample  size 

14 

7 

7 

14 

7 

7 

14 

7 

7 

14 

7 

7 

Student  sample  size 

811 

410 

401 

812 

416 

396 

764 

425 

339 

655 

327 

328 

Stayers 

t 

t 

T 

737 

377 

360 

560 

302 

258 

403 

201 

202 

New  entrants 

t 

t 

t 

75 

39 

36 

204 

123 

81 

252 

126 

126 

New  entrants  as  a percent  of 
spring  enrollment 

t 

t 

t 

9.2 

9.4 

9.1 

26.7 

28.9 

23.9 

38.5 

38.5 

38.4 

Total  leavers  (from  original  cohort) 

t 

T 

t 

74 

33 

41 

251 

108 

143 

408 

209 

199 

Leavers  as  a percent  of  fall 
2004  enrollment 

t 

t 

t 

9.1 

8.0 

10.2 

30.9 

26.3** 

35.7 

50.3 

51.0 

49.6 

Number  of  students  per  school 

(mean) 

58 

59 

57 

58 

59 

57 

55 

61 

49 

47 

47 

47 

Range  of  number  of  students  per 

school 

23-101 

38-92 

23-101 

26-103 

35-92 

26-103 

25-83 

36-83 

25-77 

25-74 

28-74 

25-65 

t Not  applicable. 

**  Treatment  group  significantly  different  from  control  group  at  the  .01  level. 
SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Implementation 

Training 

Teachers,  principals,  and  school  staff  members  in  the  intervention  schools  received  a half-day  of  program 
implementation  training  prior  to  the  beginning  of  the  first  school  year  (table  5.1,  panel  3).  The  Illinois 
research  team  provided  an  additional  half-day  of  training  to  schools  via  professional  development  days  in  fall 
2004  and  in  spring  2005,  which  also  allowed  continuing  teachers  and  staff  time  to  share  stories  and  concerns 
about  program  implementation.  In  addition,  the  principal  and  PA  coordinator  from  each  school  attended  an 
annual  meeting  in  the  winter  of  each  year  to  learn  more  about  implementing  the  schoolwide  component  of 
the  program  and  to  share  experiences. 

Data  Collection 

MPR’s  subcontractor,  Decision  Information  Resources,  Inc.  (DIR)  collected  the  child,  teacher,  and  school 
data  at  the  Illinois  site.  Table  5.3  shows  the  school  year  milestones  and  dates  of  implementation  for  the 
Illinois  site.  Data  were  collected  in  the  fall  and  spring  of  the  first  2 years  and  the  spring  of  Year  3.  The  fall 
2004  data  collection  began  on  October  12,  2004,  and  ended  on  October  28,  2004.  The  average  time  frame 
from  the  beginning  of  program  implementation  to  the  beginning  of  fall  data  collection  was  2 weeks.  As  a 
result,  initial  data  collection  took  place  after  implementation  of  the  PA  program  began.  Therefore,  these  data 
provide  a measure  of  the  students,  teachers,  and  schools  near  the  beginning  of  the  school  year,  at  a time  when 
the  PA  program  had  been  operating  for  a relatively  short  period  of  time.  The  spring  data  collection  window 
was  from  May  16,  2005,  to  June  3,  2005.  The  PA  program  had  been  implemented  for  33  weeks  at  the  time  of 
the  spring  data  collection  and  for  29  weeks  from  the  end  of  the  fall  data  collection.  Year  2 followed  a similar 
pattern,  with  implementation  occurring  at  the  start  of  the  school  year,  fall  data  collection  occurring  5 weeks 
later,  and  spring  data  collection  occurring  26  weeks  after  fall  data  collection  (and  34  weeks  after  the  start  of 
implementation).  In  spring  2007,  data  collection  occurred  35  weeks  after  the  start  of  implementation.  Data 
collection  took  from  3 to  5 weeks  at  each  collection  point. 
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Table  5.3.  Data  collection  dates — PA 


Year  1 Year  1 Year  2 Year  2 Year  3 

Data  collection  schedule  (Fall  3rd  grade)  (Spring  3rd  grade)  (Fall  4th  grade)  (Spring  4th  grade)  (Spring  5th  grade) 


School  sample  size 

14 

14 

14 

14 

14 

School  year  dates 
First  day  of  school 
Start  of  implementation 
Last  day  of  school 

9/2/04 

9/27/04 

t 

t 

t 

6/1 7/05 

9/6/05 
First  day 
t 

t 

t 

6/1 6/06 

9/5/06 
First  day 
6/1 5/07 

Data  collection 
Start 
End 

10/12/04 
1 0/28/04 

5/16/05 

6/3/05 

10/11/05 

11/2/05 

5/1/06 

5/26/06 

5/7/07 

6/1/07 

Calendar  weeks  from 
program  implementation 
to  start  of  fall  2004 
data  collection 

2 

t 

t 

t 

t 

Calendar  weeks  from  start 
of  school  to  start  of  fall 
data  collection 

6 

t 

5 

t 

t 

Calendar  weeks  from  end 
of  fall  data  collection  to 
start  of  spring  data 
collection 

t 

29 

t 

26 

t 

Calendar  weeks  from 
program  implementation 
to  start  of  spring  data 
collection 

t 

33 

t 

34 

35 

t Not  applicable. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Consent  Rates,  Completion  Rates,  and  Percentage  of  Sample  With  Data 

The  actual  number  of  student,  primary  caregiver,  and  teacher  reports  available  for  analysis  was  smaller  than 
the  number  in  the  sample  because  consent  and  completion  rates  were  below  100  percent.  Primary  caregivers 
had  to  provide  consent  before  children  could  complete  the  Child  Report,  before  their  child’s  teacher  could 
complete  the  Teacher  Report  on  Student,  and  before  they  themselves  completed  the  Primary  Caregiver 
Report.  Teachers  also  had  to  provide  consent  before  completing  the  Teacher  Report  on  Classroom  and 
School.  Of  those  with  consent,  not  all  completed  their  respective  reports.  Table  5.4  shows  the  consent  rates, 
completion  rates,  and  percentages  of  sample  with  data  for  each  of  the  four  reports  over  the  3 years.  For  the 
Child  Report  and  two  teacher  reports,  completion  rates  ranged  from  78  percent  to  100  percent,  with  no 
statistically  significant  differences  between  treatment  and  control  groups  in  consent  rates  for  any  of  these 
reports.  For  the  Teacher  Report  on  Student,  completion  rates  ranged  from  92  percent  to  100  percent,  with  a 
statistically  significant  difference  between  treatment  and  control  groups  at  two  time  points:  in  the  fall  of  2004 
and  the  spring  of  2005  the  treatment  group  teachers  completed  fewer  reports  than  the  control  teachers.  For 
the  Primary  Caregiver  Report,  the  completion  rates  dropped  over  time  from  92  percent  to  64  percent.  There 
were  no  statistically  significant  differences  between  treatment  and  control  groups  for  these  reports. 

The  percentages  of  the  sample  with  Child  Report  data  ranged  from  66  percent  to  78  percent  over  the  3 years. 
The  percentages  of  students  with  information  from  the  Teacher  Report  on  Student  ranged  from  71  percent 
to  80  percent.  The  percentages  of  students  with  data  from  the  Primary  Caregiver  Report  ranged  from  49 
percent  to  73  percent  and  declined  over  time.  The  percentages  of  teachers  with  data  from  the  Teacher  Report 
on  Classroom  and  School  ranged  from  78  percent  to  91  percent.  There  were  no  statistically  significant 
differences  between  treatment  and  control  conditions  in  percentages  of  students  with  data  for  any  of  the  four 
reports. 
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Table  5.4.  Consent  rates,  completion  rates,  and  percentage  of  sample  with  data  from  each  report — PA 


Year  1 Year  1 Year  2 Year  3 

(Fall  3rd  grade)  (Spring  3rd  grade)  (Spring  4th  grade)  (Spring  5th  grade) 

Treat-  Treat-  Treat-  Treat- 


Report 

Total 

ment 

Control 

Total 

ment 

Control 

Total 

ment  Control 

Total 

ment 

Control 

Student  sample  size 

811 

410 

401 

812 

416 

396 

764 

425 

339 

655 

327 

328 

Child  Report  (percent) 

Primary  caregiver  consent  rate 

78.5 

79.0 

78.1 

76.7 

76.4 

77.0 

72.4 

71.1 

74.0 

79.2 

80.1 

78.4 

Student  completion  rate 

94.7 

93.5 

95.8 

92.8 

93.7 

91.8 

92.0 

94.0 

89.6 

96.5 

97.7 

95.3 

Students  with  data1 

74.4 

73.9 

74.8 

71.2 

71.6 

70.7 

66.6 

66.8 

66.4 

76.5 

78.3 

74.7 

Primary  Caregiver  Report  (percent) 

Primary  caregiver  consent  rate 

78.3 

79.0 

77.6 

76.7 

76.4 

77.0 

70.9 

69.6 

72.6 

78.3 

79.5 

77.1 

Primary  caregiver  completion  rate 

92.3 

92.0 

92.6 

76.7 

77.0 

76.4 

73.6 

72.6 

74.8 

64.5 

65.0 

64.0 

Primary  caregivers  with  data1 

72.3 

72.7 

71.8 

58.9 

58.9 

58.8 

52.2 

50.6 

54.3 

50.5 

51.7 

49.4 

Teacher  Report  on  Student  (percent) 

Primary  caregiver  consent  rate2 

78.5 

79.0 

78.1 

76.7 

76.4 

77.0 

72.4 

71.1 

74.0 

79.2 

80.1 

78.4 

Teacher  completion  rate 

95.0 

92.3***  97.8 

97.4 

95.9* 

99.0 

100.0 

100.0 

100.0 

98.8 

99.2 

98.4 

Students  with  data1 

74.6 

72.9 

76.3 

74.8 

73.3 

76.3 

72.4 

71.1 

74.0 

78.3 

79.5 

77.1 

Teacher  Report  on  Classroom  and  School 
(3rd-  to  5th-grade  teachers)  (percent) 

Teacher  consent  rate 

100.0 

100.0 

100.0 

99.1 

98.4 

100.0 

100.0 

100.0 

100.0 

100.0 

100.0 

100.0 

Teacher  completion  rate 

89.5 

90.0 

88.9 

83.3 

83.3 

83.3 

86.3 

86.0 

86.7 

83.7 

78.0 

90.5 

Teachers  with  data1 

89.5 

90.0 

88.9 

82.6 

82.0 

83.3 

86.3 

86.0 

86.7 

83.7 

78.0 

90.5 

* Treatment  group  significantly  different  from  control  group  at  the  .05  level. 

***  Treatment  group  significantly  different  from  control  group  at  the  .001  level. 

1 Calculated  as  consent  rate  x completion  rate. 

2The  primary  caregiver  consent  rates  for  the  Child  Report  and  the  Teacher  Report  on  Student  are  identical,  as  the  primary  caregiver  gave  consent  to  both  together. 
SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Responses  from  students  in  the  original  cohort  (stayers)  and  new  entrants  in  the  PA  sample  were  examined  to 
investigate  possible  differences  between  the  two  groups  in  consent  rates,  completion  rates,  and  the 
percentages  of  the  sample  with  data  that  might  affect  outcome  data  (table  5.5).  In  Year  2,  new  entrants  had 
significantly  higher  consent  rates  and  percentages  of  sample  with  data  than  stayers  on  all  three  reports  (by  12 
to  17  percentage  points).  In  Year  3,  new  entrants  had  significantly  higher  completion  rates  (by  10  percentage 
points)  and  percentages  of  sample  with  data  (by  12  percentage  points)  than  stayers  for  the  Primary  Caregiver 
Report. 
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Table  5.5.  Consent  rates,  completion  rates,  and  percentage  of  sample  with  data:  Stayers  versus  new  entrants — PA 


Year  1 Year  2 Year  3 

(Spring  3rd  grade)  (Spring  4th  grade)  (Spring  5th  grade) 


Report 

Total 

Stayers 

Entrants 

Total 

Stayers 

Entrants 

Total 

Stayers 

Entrants 

Student  sample  size 

812 

737 

75 

764 

560 

204 

655 

403 

252 

Child  Report  (percent) 

Primary  caregiver  consent  rate 

76.7 

77.3 

70.7 

72.4 

68.9*** 

81.9 

79.2 

77.7 

81.7 

Student  completion  rate 

92.8 

92.5 

96.2 

92.0 

90.2* 

96.4 

96.5 

96.5 

96.6 

Students  with  data1 

71.2 

71.5 

68.0 

66.6 

62.1*** 

78.9 

76.5 

74.9 

79.0 

Primary  Caregiver  Report  (percent) 

Primary  caregiver  consent  rate 

76.7 

77.3 

70.7 

70.9 

67.0*** 

81.9 

78.3 

76.2 

81.7 

Primary  caregiver  completion  rate 

76.7 

76.1 

83.0 

73.6 

73.1 

74.9 

64.9 

60.9* 

70.9 

Primary  caregivers  with  data1 

58.9 

58.9 

58.7 

52.2 

48.9** 

61.3 

50.8 

46.4** 

57.9 

Teacher  Report  on  Student  (percent) 

Primary  caregiver  consent  rate2 

76.7 

77.3 

70.7 

72.4 

68.9*** 

81.9 

79.2 

77.7 

81.7 

Teacher  completion  rate 

97.4 

97.7 

94.3 

100.0 

100.0 

100.0 

98.8 

98.7 

99.0 

Students  with  data1 

74.8 

75.6 

66.7 

72.4 

68.9*** 

81.9 

78.3 

76.7 

81.0 

* Stayers  significantly  different  from  new  entrants  at  the  .05  level. 

**  Stayers  significantly  different  from  new  entrants  at  the  .01  level. 

***  Stayers  significantly  different  from  new  entrants  at  the  .001  level. 

1 Calculated  as  consent  rate  x completion  rate. 

2The  primary  caregiver  consent  rates  for  the  Child  Report  and  the  Teacher  Report  on  Student  are  identical,  as  the  primary  caregiver  gave  consent  to  both  together. 
SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Fidelity  of  Implementation 

Each  year,  PA’s  seven  treatment  schools  were  independently  rated  for  quantity  and  quality  of  program 
implementation  by  two  raters  from  the  research  team.  The  global  measure  of  fidelity  for  the  multisite  study 
was  used;  inter-rater  reliability  was  measured  using  Cronbach’s  alpha  (0.85  in  Year  1,  0.74  in  Year  2,  and  0.98 
in  Year  3).  The  ratings  were  combined  into  a single  consensus  rating  and  used  to  identify  schools  with  high 
implementation  fidelity.  In  Years  1 and  2,  four  treatment  schools  were  identified  as  having  high  fidelity,  and 
in  Year  3,  three  treatment  schools  were  identified  as  having  high  fidelity.  Cohen’s  kappa  was  used  as  the 
measure  of  agreement  when  identifying  schools  as  high  fidelity,  and  it  equaled  0.72  in  Year  1,  0.46  in  Year  2, 
and  1.00  in  Year  3. 

Initial  Characteristics 

This  section  describes  the  initial  characteristics  of  the  students,  teachers,  and  schools  participating  in  the 
evaluation  of  the  PA  program.  These  characteristics  were  collected  from  students  who  were  enrolled  in  the 
third  grade  at  the  study  schools  in  fall  2004,  as  well  as  from  their  primary  caregivers  and  third-grade  teachers. 
In  addition,  third-,  fourth-,  and  fifth-grade  teachers  and  principals  in  the  study  schools  provided  information 
about  activities  related  to  social  and  character  development  in  these  schools.  Documenting  the  characteristics 
of  students,  teachers,  and  schools  and  initial  measures  of  key  outcomes  at  a point  before  the  interventions 
had  been  operating  for  an  extended  period  helps  to  determine  whether  the  random  assignment  of  schools  to 
treatment  and  control  status  produced  treatment  and  control  groups  with  similar  distributions  of  observed 
characteristics.  As  noted  in  the  following  discussion,  3 significant  differences  (out  of  62  comparisons,  with  3 
expected  to  be  significant  by  chance)  were  found  between  the  observed  characteristics  of  the  treatment  and 
control  students,  teachers,  and  schools;  all  3 reflected  the  use  of  SACD  activities  in  the  classroom  and  school. 

Characteristics  of  Children,  Their  Families,  and  Communities 

There  were  no  significant  differences  between  the  treatment  and  control  groups  in  the  observed  student, 
caregiver,  and  community  characteristics  (table  5.6).  For  students,  the  mean  age  was  8.3  years.  The  sample 
contained  roughly  equal  percentages  of  girls  (54%)  and  boys  (46%).  The  sample  was  ethnically  diverse,  with 
Black  non-Hispanic  students  making  up  the  majority  of  the  sample  (51%).  White  non -Hispanic  students 
made  up  6 percent  of  the  sample  and  Hispanic  students  made  up  37  percent  of  the  sample. 

The  sample  was  diverse  in  its  levels  of  family  income,  education  levels  of  primary  caregivers  of  the  children  in 
the  sample,  and  family  situation.  For  the  total  sample,  66  percent  of  the  children  lived  in  a household  where 
the  income  was  1 35  percent  of  the  federal  poverty  level  or  lower,  which  is  the  income  threshold  for  eligibility 
for  free  school  meals.  More  than  one-quarter  (27%)  of  primary  caregivers  had  not  completed  high  school. 
Fewer  than  half  of  the  children  (47%)  lived  with  both  their  mother  and  their  father.  There  were  no  significant 
differences  between  the  treatment  and  control  groups  in  these  characteristics. 

The  mean  values  of  the  outcomes  for  children’s  behavior  and  attitudes  as  reported  by  the  primary  caregiver, 
child,  and  teacher  at  initial  data  collection  in  fall  2004  are  shown  in  table  5.7.  There  were  no  significant 
differences  between  the  treatment  and  control  groups  in  these  scores. 
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Table  5.6.  Initial  characteristics  of  children,  their  families,  and  communities — PA 


Characteristic 

Total 

Treatment 

Control 

Student  sample  size 

586 

298 

288 

Student  demographics 

Gender  (percent) 

Male 

46.3 

47.8 

44.7 

Female 

53.7 

52.2 

55.3 

Race/ethnicity  (percent) 

White  (non-Hispanic) 

6.0 

5.0 

7.0 

Black  (non-Hispanic) 

51.1 

51.9 

50.2 

Hispanic 

37.4 

39.2 

35.5 

Other 

5.5 

3.8 

7.3 

Age  (in  years)  (mean) 

8.3 

8.3 

8.3 

Primary  caregiver  and  family  characteristics 

Primary  caregiver’s  age  (in  years)  (mean) 

35.8 

35.3 

36.3 

Primary  caregiver’s  race/ethnicity  (percent) 

White  (non-Hispanic) 

8.2 

7.3 

9.1 

Black  (non-Hispanic) 

51.2 

52.2 

50.2 

Hispanic 

35.5 

36.9 

34.0 

Other 

5.2 

3.6 

6.7 

Primary  caregiver’s  education  (percent) 

Did  not  complete  high  school 

27.4 

26.3 

28.5 

Completed  high  school  or  equivalent 

30.9 

27.1 

34.7 

Some  college 

35.7 

38.5 

32.9 

Bachelor’s  or  higher  degree 

6.1 

8.1 

4.0 

Primary  caregiver’s  employment  (percent) 

Full-time 

45.4 

44.1 

46.7 

Other 

54.6 

55.9 

53.3 

Primary  caregiver’s  marital  status  (percent) 

Married 

41.4 

40.6 

42.2 

Other 

58.6 

59.4 

57.8 

Students  who  live  in  one  household  (percent) 

92.7 

93.5 

92.0 

Number  of  individuals  in  household  (mean) 

5.1 

4.9 

5.3 

Primary  caregiver’s  relationship  to  child  (percent) 

Mother  (stepmother) 

83.2 

84.3 

82.1 

Father  (stepfather) 

10.2 

8.7 

11.6 

Other  relative/nonrelative 

6.6 

7.0 

6.3 

See  notes  at  end  of  table. 
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Table  5.6.  Initial  characteristics  of  children,  their  families,  and  communities — PA — Continued 


Characteristic 

Total 

Treatment 

Control 

Student  lives  with  (percent) 

Mother  (stepmother)  and  father  (stepfather) 

46.6 

44.1 

49.1 

Mother  (stepmother)  only;  father  (stepfather)  not  present 

45.3 

47.7 

42.8 

Father  (stepfather)  only;  mother  (stepmother)  not  present 

3.0 

2.7 

3.4 

Other  relative/nonrelative,  parents  not  present 

5.1 

5.5 

4.8 

Highest  education  of  anyone  in  household  (percent) 

Did  not  complete  high  school 

18.8 

17.1 

20.5 

Completed  high  school  or  equivalent 

31.1 

29.7 

32.5 

Some  college 

39.7 

42.3 

37.1 

Bachelor’s  or  higher  degree 

10.4 

11.0 

9.9 

Total  household  income  (percent) 

Less  than  $20,000 

55.4 

53.6 

57.2 

$20,000  to  $39,999 

28.1 

29.4 

26.7 

$40,000  to  $59,999 

10.2 

10.7 

9.8 

$60,000  or  more 

6.3 

6.3 

6.4 

Income-to-poverty-threshold  ratio — Below  135  percent  (percent) 

65.7 

60.6 

70.7 

Income-to-poverty-threshold  ratio — 135  to  185  percent  (percent) 

18.5 

20.8 

16.3 

Income-to-poverty-threshold  ratio — Above  185  percent  (percent) 

15.8 

18.6 

12.9 

Alabama  Parenting  Questionnaire — Poor  Monitoring  and 

Supervision  Subscale  (mean) 

1.2 

1.2 

1.2 

Alabama  Parenting  Questionnaire — Positive  Parenting 

Subscale  (mean) 

3.5 

3.5 

3.5 

Confusion,  Hubbub,  and  Order  Scale  (mean) 

2.2 

2.1 

2.2 

Community  characteristics  (mean) 

Community  Risks  Scale 

1.9 

1.8 

1.9 

Community  Resources  Scale 

2.7 

2.7 

2.6 

Child-Centered  Social  Control  Scale 

2.8 

2.8 

2.8 

NOTE:  No  statistically  significant  differences  were  found  between  values  for  treatment  and  control  groups.  Weights,  which  assign 
equal  weight  to  each  school  within  the  program,  were  used  in  producing  the  treatment,  control,  and  overall  means.  Statistical  tests 
were  conducted  using  regressions  that  included  program  indicators  to  account  for  the  sample  design  and  adjusted  for  clustering  at 
the  school  level. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Table  5.7.  Mean  scores  and  standard  deviations  for  initial  outcome  measures  of  sample — PA 


Outcome  measure-Report 

Range 

Total 

Mean 

SD 

Treatment 
Mean  SD 

Control 
Mean  SD 

Social  and  Emotional  Competence  Domain 

Self-Efficacy  for  Peer  Interaction-CR 

1-4 

2.8 

0.6 

2.9 

0.7 

2.8 

0.6 

Normative  Beliefs  About  Aggression-CR 

1-4 

1.3 

0.4 

1.3 

0.5 

1.2 

0.4 

Empathy-CR 

1-3 

2.4 

0.4 

2.4 

0.4 

2.4 

0.4 

Behavior  Domain 

Altruistic  Behavior-CR 

0-3 

1.6 

0.8 

1.6 

0.8 

1.7 

0.7 

Altruistic  Behavior-TRS 

1-4 

1.4 

0.5 

1.5 

0.6 

1.3 

0.4 

Altruistic  Behavior-PCR 

1-4 

2.4 

0.7 

2.4 

0.7 

2.5 

0.8 

Positive  Social  Behavior-TRS 

1-4 

2.9 

0.7 

2.9 

0.7 

2.9 

0.7 

Positive  Social  Behavior-PCR 

1-4 

2.9 

0.5 

2.9 

0.5 

2.9 

0.5 

Problem  Behavior-CR 

0-3 

0.3 

0.5 

0.4 

0.5 

0.3 

0.4 

Problem  Behavior-TRS 

1-4 

1.4 

0.5 

1.4 

0.4 

1.5 

0.5 

Problem  Behavior-PCR 

1-4 

1.6 

0.4 

1.5 

0.3 

1.6 

0.4 

ADHD-Related  Behavior-TRS 

1-4 

1.7 

0.6 

1.7 

0.6 

1.8 

0.6 

Academics  Domain 

Academic  Competence  and  Motivation-TRS 

1-5 

2.7 

0.9 

2.7 

0.9 

2.7 

0.9 

Engagement  with  Learning-CR 

1-4 

3.5 

0.7 

3.5 

0.7 

3.5 

0.8 

Perceptions  of  School  Climate  Domain 

Positive  School  Orientation-CR 

1-4 

3.1 

0.7 

3.1 

0.7 

3.1 

0.7 

Negative  School  Orientation-CR 

1-4 

2.1 

0.7 

2.2 

0.6 

2.1 

0.7 

Student  Afraid  at  School-CR 

1-4 

2.6 

0.9 

2.6 

0.9 

2.6 

0.9 

Victimization  at  School-CR 

0-3 

0.9 

0.8 

0.9 

0.8 

0.9 

0.8 

Student  sample  size — PCR 

586 

298 

288 

Student  sample  size — CR 

603 

303 

300 

Student  sample  size — TRS 

605 

299 

306 

NOTE:  Abbreviations  are 
CR:  Child  Report 
PCR:  Primary  Caregiver  Report 
TRS:  Teacher  Report  on  Student 
ADHD:  Attention  deficit  hyperactivity  disorder 
SD:  Standard  deviation 

No  statistically  significant  differences  were  found  between  values  for  treatment  and  control  groups.  Weights,  which  assign  equal 
weight  to  each  school  within  the  program,  were  used  in  producing  the  treatment,  control,  and  overall  means.  Statistical  tests  were 
conducted  using  regressions  that  included  program  indicators  to  account  for  the  sample  design  and  adjusted  for  clustering  at  the 
school  level.  Sample  size  may  differ  for  some  outcomes  due  to  nonresponse. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Characteristics  of  Teachers  and  Schools 

Table  5.8  describes  the  third-,  fourth-,  and  fifth-grade  teachers  at  the  study  schools.  Slighdy  fewer  than  half 
were  White  non- Hispanic  (49%).  Most  of  the  teachers  were  female  (85%)  and  had  an  average  of  15  years  of 
total  teaching  experience.  Slighdy  fewer  than  half  (47%)  held  an  advanced  or  specialist  degree.  There  were  no 
statistically  significant  differences  between  the  treatment  and  control  groups  of  teachers. 

Data  regarding  school  characteristics  were  drawn  from  the  Common  Core  of  Data  in  order  to  compare 
treatment  and  control  schools.  There  were  no  significant  differences  between  the  two  groups  of  schools  in 
terms  of  student  composition  (race/ ethnicity  and  school  lunch  eligibility),  number  of  students  enrolled, 
number  of  full-time  teachers,  Title  1 status,  or  number  of  years  the  principal  had  been  at  the  school  (table 
5.9).  In  addition,  there  were  no  significant  differences  between  treatment  and  control  schools  in  terms  of 
location  (urban,  suburban,  or  rural)  or  lowest  and  highest  grade  offered  (these  data  are  not  shown  in  a table). 

Table  5.8.  Initial  characteristics  of  teachers  in  sample — PA 


Characteristic 

Total 

Treatment 

Control 

Teacher  sample  size 

102 

54 

48 

Gender  (percent) 

Male 

14.9 

22.2 

7.5 

Female 

85.1 

77.8 

92.5 

Race/ethnicity  (percent) 

White  (non-Hispanic) 

49.1 

40.5 

57.7 

Other 

50.9 

59.5 

42.3 

Number  of  years  teaching  experience  (mean) 

15.0 

15.7 

14.3 

Number  of  years  teaching  experience  in  this  school  (mean) 

9.0 

10.0 

8.1 

Type  of  teaching  certificate  (percent) 

Regular  state  certificate  or  advanced  professional  certificate 

90.5 

92.9 

88.1 

Other 

9.5 

7.1 

11.9 

Education  (percent) 

Bachelor’s  degree 

53.1 

57.1 

49.1 

Advanced  degree/other 

46.9 

42.9 

50.9 

NOTE:  No  statistically  significant  differences  were  found  between  values  for  treatment  and  control  groups.  Weights,  which  assign 
equal  weight  to  each  school  within  the  program,  were  used  in  producing  the  treatment,  control,  and  overall  means.  Statistical  tests 
were  conducted  using  regressions  that  included  program  indicators  to  account  for  the  sample  design  and  adjusted  for  clustering  at 
the  school  level.  Sample  size  may  differ  for  some  outcomes  due  to  nonresponse. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 


316 


Chapter  5.  Positive  Action 


Table  5.9.  Initial  characteristics  of  schools  in  sample — PA 


Characteristic 

Total 

Treatment 

Control 

School  sample  size 

14 

7 

7 

Student  race/ethnicity  (percent) 

White  (non-Hispanic) 

9.2 

9.1 

9.4 

Black  (non-Hispanic) 

55.1 

53.6 

56.5 

Hispanic 

31.9 

32.8 

31.0 

Other 

3.8 

4.5 

3.1 

Students  eligible  for  free  or  reduced-price  lunch  (percent) 

90.6 

92.6 

88.7 

Number  of  students  enrolled  (mean) 

532.0 

521.1 

542.9 

Number  of  full-time  teachers  (mean) 

30.0 

28.5 

331.5 

Title  1 status  (percent) 

Title  1 eligible  school 

100.0 

100.0 

100.0 

Schoolwide  Title  1 

100.0 

100.0 

100.0 

Number  of  years  principal  has  been  at  this  school  (mean) 

5.9 

5.9 

5.9 

NOTE:  No  statistically  significant  differences  were  found  between  values  for  treatment  and  control  groups.  Weights,  which  assign 
equal  weight  to  each  school  within  the  program,  were  used  in  producing  the  treatment,  control,  and  overall  means.  Statistical  tests 
were  conducted  using  regressions  that  included  program  indicators  to  account  for  the  sample  design  and  adjusted  for  clustering  at 
the  school  level.  Sample  size  may  differ  for  some  outcomes  due  to  nonresponse. 

SOURCE:  NCES  Common  Core  of  Data  (2003-04),  the  Social  and  Character  Development  (SACD)  Research  Program. 


In  the  Teacher  Report  on  Classroom  and  School,  teachers  reported  on  nine  dimensions  of  school 
environment  (these  data  are  not  shown  in  a table):  feelings  of  safety,  adequacy  of  resources,  student  support, 
freedom  to  teach  as  desired,  affiliation  with  and  ties  to  colleagues,  innovation  regarding  new  approaches  to 
teaching,  professional  interest,  participatory  decisionmaking,  and  work  pressure.  There  were  no  statistically 
significant  differences  between  treatment  and  control  schools  in  these  reports. 

The  Level  of  SACD  in  the  Schools  Near  the  Beginning  of  the  Study 

During  the  initial  data  collection,  principals  and  teachers  reported  on  the  SACD  activities  used  in  the  schools 
and  classrooms,  the  availability  of  SACD  materials,  and  the  professional  development  provided  on  SACD. 
Table  5.10  shows  that  the  majority  of  the  principals  reported  activities  to  promote  six  social  and  character 
development  goals:  violence  prevention  and  peace  promotion  (100%),  social  and  emotional  development 
(93%),  character  education  (93%),  tolerance  and  diversity  (93%),  risk  prevention  and  health  promotion  (93%), 
and  civic  responsibility  and  community  service  (100%).  In  addition,  86  percent  of  the  principals  reported  the 
use  of  activities  directed  toward  behavior  management.  There  were  no  statistically  significant  differences 
between  the  treatment  group  and  the  control  group  in  the  percentages  on  principal  reports,  although  this  may 
be  due  to  the  small  principal  sample  size.  The  percentages  of  teachers  reporting  the  use  of  these  activities  in 
their  classrooms  ranged  from  45  percent  to  81  percent,  and  there  were  no  significant  differences  between 
treatment  and  control  teachers.  With  respect  to  the  use  of  schoolwide  activities,  52  percent  to  76  percent  of 
teachers  reported  that  their  schools  used  such  activities.  There  was  a significant  difference  between  treatment 
and  control  teachers  on  reports  of  their  use  of  unspecified  schoolwide  SACD  activities  (one  out  of  six 
comparisons),  with  treatment  teachers  reporting  fewer  “other  activities”  than  control  teachers  (values 
suppressed  to  protect  confidentiality). 
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Table  5.10.  Principal  and  teacher  initial  reports  on  use  of  SACD  programs  or  activities  in 
sample — PA 


SACD  program  or  activity 

Total 

Treatment 

Control 

Principal  sample  size 

14 

7 

7 

Teacher  sample  size 

102 

54 

48 

Principals  reporting  that  school  had  programs  or  activities 
to  promote  the  following  SACD  goals  (percent) 

Violence  prevention  and  peace  promotion 

100.0 

100.0 

100.0 

Social  and  emotional  development 

92.9 

100.0 

85.7 

Character  education 

92.9 

85.7 

100.0 

Tolerance  and  diversity 

92.9 

100.0 

85.7 

Risk  prevention  and  health  promotion 

92.9 

85.7 

100.0 

Civic  responsibility  and  community  service 

100.0 

100.0 

100.0 

Behavior  management 

85.7 

100.0 

71.4 

None  of  the  above 

0.0 

0.0 

0.0 

Teachers  reporting  on  using  programs  or  activities  in  their 
class  to  promote  the  following  SACD  goals  (percent) 

Violence  prevention  and  peace  promotion 

63.0 

60.2 

65.8 

Social  and  emotional  development 

70.8 

70.2 

71.3 

Character  education 

81.0 

79.4 

82.5 

Tolerance  and  diversity 

57.7 

60.7 

54.8 

Risk  prevention  and  health  promotion 

54.6 

58.8 

50.4 

Civic  responsibility  and  community  service 

44.6 

45.8 

43.4 

Behavior  management 

77.5 

76.1 

79.0 

None  of  the  above 

6.9 

6.0 

7.7 

Teachers  reporting  schoolwide  use  of  the  following 
activities  to  promote  SACD  (percent) 

Morning  announcements  or  videos 

60.8 

62.0 

59.6 

School  assemblies 

75.7 

70.4 

81.0 

School  newspapers  or  bulletins 

67.1 

61.8 

72.4 

Special  school  days 

52.3 

52.9 

51.7 

Special  events 

67.0 

65.5 

68.4 

Other  activities 

11.8 

r 

t 

J Reporting  standards  not  met.  Values  suppressed  to  protect  confidentiality. 

* Treatment  group  significantly  different  from  control  group  at  the  .05  level. 

NOTE:  Weights,  which  assign  equal  weight  to  each  school  within  each  of  the  programs  and  to  each  program  across  programs,  were 
used  in  producing  the  treatment,  control,  and  overall  means.  Statistical  tests  were  conducted  using  regressions  that  included 
program  indicators  to  account  for  the  sample  design  and  adjusted  for  clustering  at  the  school  level.  Sample  size  may  differ  for  some 
outcomes  due  to  nonresponse. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Teachers  reported  using  a broad  range  of  teaching  materials  to  support  SACD  activities  (table  5.11),  including 
teacher  guides  (70%),  student  materials  (68%),  instructional  aids  (38%),  giveaways  (52%),  and  children’s 
literature  (40%).  There  were  no  statistically  significant  differences  between  treatment  and  control  teachers  in 
their  use  of  these  materials. 

Teachers  also  reported  using  a wide  variety  of  teaching  strategies  (table  5.11).  Nearly  all  teachers  (99%) 
reported  using  any  of  the  20  strategies  asked  about,  and  teachers  used  an  average  of  12.1  of  the  strategies. 
There  were  significant  differences  between  treatment  and  control  teacher  reports  on  the  use  of  2 of  the  20 
strategies.  One  was  a significant  difference  in  peer  mediation,  with  treatment  teachers  using  this  strategy  more 
often  than  control  teachers  (54%  versus  35%).  The  other  was  a significant  difference  in  the  use  of  time  out 
for  negative  behavior,  with  treatment  teachers  using  this  strategy  less  often  than  control  teachers  (72%  versus 
87%). 
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Table  5.11.  Teacher  initial  reports  on  use  of  SACD  materials  and  classroom  strategies  in 
sample — PA 


SACD  material  and  classroom  strategy 

Total 

Treatment 

Control 

Teacher  sample  size 

102 

54 

48 

Teachers  using  the  following  materials  in  conjunction  with 
social  and  character  development  activities  (percent) 

Teacher  guides  (manuals,  curricula) 

69.7 

76.2 

63.2 

Student  materials  (workbooks,  worksheets) 

67.5 

76.3 

58.6 

Instructional  aids  (games,  software,  videos) 

37.6 

36.8 

38.5 

Giveaways  (bookmarks,  stickers) 

52.4 

56.3 

48.4 

Children’s  literature 

39.8 

36.3 

43.2 

Other  types  of  materials 

14.1 

5.7 

22.5 

Do  not  use  any  of  the  materials  listed  above 

8.2 

9.6 

6.8 

Teachers  using  any  of  the  strategies  listed  below  to  promote 
social  and  character  development  in  the  classroom  (percent) 

99.2 

98.4 

100.0 

Number  of  strategies  (listed  below)  used  by  teachers  to  promote 
social  and  character  development  in  the  classroom  (mean) 

12.1 

12.3 

12.0 

Teachers  using  each  of  the  following  strategies  to  promote 
social  and  character  development  (percent) 

Role-playing 

71.6 

71.3 

71.9 

Cooperative  learning 

98.3 

98.4 

98.2 

Peer  group  discussions 

86.6 

89.2 

84.0 

Direct  instruction  of  social  and  character  development 

83.4 

86.3 

80.5 

Skill  training 

41.2 

40.0 

42.5 

Incorporating  social  and  character  development  into 
academic  curriculum 

78.6 

81.4 

75.9 

Parent  training 

3.2 

0.0 

6.4 

Parent/community  involvement  in  program 
development  or  delivery 

16.3 

17.4 

15.2 

Mentoring 

36.3 

27.5 

45.1 

Good  behavior  notes  sent  home  daily  or  weekly 

66.6 

67.0 

66.1 

Presenting  role  models 

60.6 

53.5 

67.7 

Targeted  story  reading  or  writing  on  SACD  themes 

84.2 

85.9 

82.6 

Peer  mediation 

44.7 

54.4* 

35.1 

Honor  roll  for  positive  behavior 

64.3 

64.5 

64.1 

Pledges  or  recitations  on  social  and  character 
development  themes 

47.1 

50.7 

43.6 

Guided  visualization 

54.8 

57.5 

52.1 

Student-led/student-assisted  instruction 

50.8 

55.5 

46.2 

Journaling 

82.4 

82.0 

82.9 

Time  out  for  negative  behavior 

79.7 

72.1* 

87.4 

Daily  or  weekly  rewards  for  positive  behavior 

93.8 

91.2 

96.4 

* Treatment  group  significantly  different  from  control  group  at  the  .05  level. 

NOTE:  Weights,  which  assign  equal  weight  to  each  school  within  each  of  the  programs  and  to  each  program  across  programs,  were 
used  in  producing  the  treatment,  control,  and  overall  means.  Statistical  tests  were  conducted  using  regressions  that  included 
program  indicators  to  account  for  the  sample  design  and  adjusted  for  clustering  at  the  school  level.  Sample  size  may  differ  for  some 
outcomes  due  to  nonresponse. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Principals  and  teachers  reported  on  participation  in  and  amount  of  SACD  training  and  staff  development 
provided  over  the  previous  12  months  (table  5.12).  Principals  reported  higher  participation  rates  (93%  versus 
68%)  than  did  teachers,  although  principals  and  teachers  reported  virtually  the  same  number  of  training  hours 
(5.0  versus  5.2).  There  were  no  significant  differences  between  treatment  and  control  principals  or  teachers 
on  any  of  these  measures. 


Table  5.12.  Principal  and  teacher  initial  reports  on  SACD  professional  development  in  sample — 
PA 


SACD  professional  development 

Total 

Treatment 

Control 

Principal  sample  size 

14 

7 

7 

Teacher  sample  size 

102 

54 

48 

Principals  reporting  that  staff  participated  in  social  and  character 
development  training  within  the  past  year  (percent) 

92.9 

100.0 

85.7 

Teachers  reporting  participation  in  social  and  character 
development  training  within  the  past  12  months  (percent) 

67.5 

72.4 

62.6 

Number  of  hours  of  social  and  character  development  training 
principals  report  were  provided  to  each  staff  person  last  year  (mean) 

5.0 

6.2 

4.1 

Number  of  hours  of  social  and  character  development  training 
teachers  report  receiving  during  the  past  12  months  (mean) 

5.2 

5.6 

4.9 

Teachers  reporting  receiving  training  in  the  past  12  months 
in  the  following  areas  (percent) 

Violence  prevention  and  peace  promotion 

17.1 

16.0 

18.2 

Social  and  emotional  development 

22.3 

28.9 

15.7 

Character  education 

35.6 

41.0 

30.2 

Tolerance  and  diversity 

15.5 

16.0 

14.9 

Risk  prevention  and  health  promotion 

6.7 

t 

t 

Civic  responsibility  and  community  service 

7.3 

8.6 

6.0 

Behavior  management 

37.2 

33.3 

41.2 

$ Reporting  standards  not  met.  Values  suppressed  to  protect  confidentiality. 

NOTE:  No  statistically  significant  differences  were  found  between  values  for  treatment  and  control  groups.  Weights,  which  assign 
equal  weight  to  each  school  within  each  of  the  programs  and  to  each  program  across  programs,  were  used  in  producing  the 
treatment,  control,  and  overall  means.  Statistical  tests  were  conducted  using  regressions  that  included  program  indicators  to 
account  for  the  sample  design  and  adjusted  for  clustering  at  the  school  level.  Sample  size  may  differ  for  some  outcomes  due  to 
nonresponse. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 


The  data  on  the  initial  level  of  SACD  activity  emphasize  that  the  control  condition  was  a “standard  practice” 
control.  Standard  practice  at  the  control  schools  included  using  SACD  activities,  materials,  and  practices, 
along  with  professional  development  for  staff,  at  rates  and  in  types  and  amounts  similar  to  the  treatment 
schools.  For  example,  the  percentages  of  teachers  who  reported  using  programs  or  activities  to  promote 
specific  SACD  goals  ranged  from  46  percent  to  79  percent  in  the  treatment  schools  and  from  43  percent  to 
83  percent  in  the  control  schools.  The  3 significant  differences  between  the  treatment  and  control  conditions 
in  the  use  of  SACD  activities  was  the  number  expected  by  chance  (3  out  of  62  comparisons);  2 of  these 
differences  favored  the  control  group  and  1 favored  the  treatment  group. 
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Impacts  on  Use  of  SACD  Activities 

The  introduction  of  the  formal  PA  program  would  be  expected  to  increase  the  use  of  SACD  activities  in  the 
treatment  schools  in  comparison  to  the  control  schools.  The  analysis  of  this  impact  is  based  on  the  Teacher 
Report  on  Classroom  and  School  (TRCS).  Every  spring,  third-,  fourth-,  and  fifth-grade  teachers  provided 
information  through  the  TRCS  about  the  social  and  character  development  activities  they  used  in  their 
classrooms.  Specifically,  information  from  the  TRCS  was  used  to  determine  difference  between  treatment  and 
control  teachers  in  several  areas: 

1 . the  use  of  SACD  activities  in  their  classrooms  overall  and  by  SACD  goal; 

2.  the  use  of  materials  and  strategies  to  implement  the  SACD  activities  within  classrooms  and  within 
the  entire  school; 

3.  the  use  of  staff  development  to  support  the  teachers;  and 

4.  teacher  support  for  SACD  efforts  in  the  school  and  the  use  of  practices  conducive  to  the  social  and 
character  development  of  students. 

TRCS  consent  and  completion  rates  (table  5.4)  led  to  78  percent  to  91  percent  of  all  teachers  having  data  for 
the  3 years.  To  estimate  intervention  impacts  for  each  of  the  outcomes,  testing  of  the  statistical  significance  of 
the  differences  in  means  was  used.  Preliminary  analysis  indicated  little  or  no  gains  in  precision  from  using 
covariates.  Before  testing  the  mean  differences,  the  data  were  weighted  such  that  each  school  received  equal 
weight.  Standard  errors  of  the  impact  estimates  account  for  the  clustering  of  teachers  within  schools.  In 
addition,  a set  of  heuristics  (described  in  chapter  1)  was  applied  to  determine  whether  each  outcome  domain 
was  statistically  significant  after  adjustments  were  made  for  the  multiple  tests  conducted. 

Use  of  Activities 

The  percentages  of  control  teachers  reporting  using  any  SACD  activities  in  their  classroom  ranged  from  83 
percent  to  95  percent  over  the  3 years  (table  5.13,  panel  1).  For  the  six  individual  SACD  goals,  the  range 
varied  from  38  percent  to  71  percent  in  Year  1,  52  percent  to  77  percent  in  Year  2,  and  53  percent  to  65 
percent  in  Year  3.  Control  teachers’  use  of  behavior  management  activities  ranged  from  76  percent  to  82 
percent  over  this  period.  The  percentages  of  control  teachers  who  reported  using  any  SACD  activities  in  their 
classrooms  for  at  least  1 hour  per  week  (panel  2)  ranged  from  36  percent  to  77  percent  over  the  3 years.  For 
the  six  individual  SACD  goals,  the  ranges  varied  from  13  percent  to  26  percent  in  Year  1,  23  percent  to  44 
percent  in  Year  2,  and  19  percent  to  31  percent  in  Year  3.  Their  use  of  behavior  management  activities  ranged 
from  42  percent  to  69  percent  over  this  period.  These  findings  show  that  the  control  schools  were  using  these 
activities  as  part  of  their  standard  practice  related  to  social  and  character  development. 

For  teachers’  reported  use  of  any  SACD  activity,  48  comparisons  were  made,  with  2 expected  to  be 
significant  by  chance.  There  were  3 significant  impacts  in  use  of  specific  SACD  activities  (panels  1 and  2).  In 
Years  2 and  3,  more  treatment  teachers  than  control  teachers  reported  using  social  and  emotional 
development  activities  (impact  = 26  and  30  percentage  points),  and  in  Year  3 more  treatment  teachers 
reported  using  character  education  activities  (impact  = 32  percentage  points).  More  treatment  teachers  than 
control  teachers  reported  using  any  activity  for  at  least  1 hour  per  week  in  Year  1 (impact  = 41  percentage 
points).  More  treatment  teachers  also  reported  using  specific  activities  for  at  least  1 hour  per  week:  violence 
prevention  and  peace  promotion  in  Years  1 and  3 (impact  = 27  and  26  percentage  points),  social  and 
emotional  development  in  all  years  (impact  = 37,  29,  and  44  percentage  points),  character  education  in  Years 
1 and  3 (impact  = 41  and  58  percentage  points),  and  tolerance  and  diversity  in  Year  3 (impact  = 27 
percentage  points).  After  the  heuristics  were  applied,  the  domain  for  engagement  in  SACD  activities  showed 
that  the  PA  program  had  statistically  significant  impacts  in  Years  1 and  3. 

For  teachers’  reported  use  of  any  named  SACD  activity  (panels  3 and  4),  42  comparisons  were  made,  with  2 
expected  to  be  significant  by  chance.  Six  of  the  12  impact  estimates  in  Year  1,  8 of  the  12  in  Year  2,  and  8 of 
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the  12  in  Year  3 were  statistically  significant.  In  Year  1,  impacts  occurred  on  social  and  emotional 
development  (impact  = 39  percentage  points,  and  impact  = 35  percentage  points  for  at  least  1 hour  per 
week),  character  education  (impact  = 47  percentage  points,  and  impact  = 44  percentage  points  for  at  least  1 
hour  per  week),  and  any  named  activity  (impact  = 47  percentage  points,  and  impact  = 47  percentage  points 
for  at  least  1 hour  per  week).  In  Year  2,  impacts  were  seen  for  social  and  emotional  development  (impact  = 
41  percentage  points,  and  impact  = 37  percentage  points  for  at  least  1 hour  per  week),  character  education 
(impact  = 45  percentage  points),  tolerance  and  diversity  (impact  = 32  percentage  points),  risk  prevention  and 
health  promotion  (impact  = 35  percentage  points,  and  impact  = 30  percentage  points  for  at  least  1 hour  per 
week),  civic  responsibility  and  community  service  (impact  =19  percentage  points),  and  any  named  activity  for 
at  least  1 hour  per  week  (impact  = 36  percentage  points).  In  Year  3,  there  were  significant  impacts  on  the  use 
of  named  violence  prevention  and  peace  promotion  activities  (impact  = 28  percentage  points)  for  at  least  1 
hour  per  week,  social  and  emotional  activities  (impact  = 37  percentage  points,  impact  = 34  percentage  points 
at  least  1 hour  per  week),  character  education  (impact  = 49  percentage  points,  impact  = 46  percentage  points 
for  at  least  1 hour  per  week),  tolerance  and  diversity  (impact  = 38  percentage  points,  impact  =31  percentage 
points  for  at  least  1 hour  per  week),  and  risk  prevention  and  health  promotion  (impact  = 26  percentage 
points).  The  overall  impact  of  the  PA  program  on  the  domain  for  engagement  in  named  SACD  activities  was 
significant  in  all  3 years. 
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Table  5.13.  Impacts  on  teacher-reported  SACD  classroom  activities — PA 

Panel  1 : Engagement  in  any  activities  to  promote  SACD  goals1 

Year  1 Year  2 Year  3 

(Spring  3rd  grade) (Spring  4th  grade) (Spring  5th  grade) 

T reat-  T reat-  T reat- 


SACD  activity 

ment 

Control 

Impact 

p-value 

ment 

Control 

Impact 

p-value 

ment 

Control 

Impact 

p-value 

Teacher  sample  size 

60 

54 

43 

39 

39 

38 

Violence  prevention  and  peace  promotion 
(percent) 

Social  and  emotion  development 

66.1 

58.5 

7.6 

0.523 

71.6 

77.4 

-5.8 

0.594 

68.2 

60.7 

7.5 

0.590 

(percent) 

67.7 

54.7 

13.0 

0.259 

88.7* 

62.6 

26.1 

0.013 

83.9* 

53.6 

30.3 

0.019 

Character  education  (percent) 

88. 9A 

70.9 

18.0 

0.086 

91 .1 A 

75.2 

15.9 

0.071 

94.4* 

62.4 

32.0 

0.017 

Tolerance  and  diversity  (percent) 

Risk  prevention  and  health  promotion 

59.1 A 

37.7 

21.5 

0.086 

66.1 

64.1 

2.0 

0.893 

77.1 

53.1 

24.1 

0.165 

(percent) 

Civic  responsibility  and  community  service 

62.0 

47.5 

14.5 

0.128 

66.8 

70.9 

-4.1 

0.769 

67.1 

61.6 

5.6 

0.748 

(percent) 

43.7 

46.2 

-2.4 

0.815 

55.4 

51.7 

3.6 

0.818 

53.5 

65.1 

-11.6 

0.325 

Any  SACD  goal  (percent) 

91.8 

82.6 

9.1 

0.334 

91.1 

94.8 

-3.6 

0.494 

96.4 

88.8 

7.6 

0.325 

Behavior  management  (percent) 

80.9 

81.6 

-0.8 

0.937 

91.7 

76.3 

15.3 

0.139 

81.2 

81.1 

0.1 

0.995 

See  note  at  end  of  table. 
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Table  5.13.  Impacts  on  teacher-reported  SACD  classroom  activities — PA — Continued 

Panel  2:  Engagement  in  any  activities  to  promote  SACD  goals  for  at  least  1 hour  per  week 

Year  1 Year  2 Year  3 

(Spring  3rd  grade) (Spring  4th  grade) (Spring  5th  grade) 

T reat-  T reat-  T reat- 


SACD  activity 

ment 

Control 

Impact 

p-value 

ment 

Control 

Impact 

p-value 

ment 

Control 

Impact 

p-value 

Teacher  sample  size 

60 

54 

43 

39 

39 

38 

Violence  prevention  and  peace  promotion 
(percent) 

43.6* 

17.1 

26.5 

0.021 

46.4 

31.4 

14.9 

0.209 

48.1* 

22.1 

26.0 

0.028 

Social  and  emotional  development 
(percent) 

51.3* 

14.7 

36.7 

0.011 

57.2* 

28.2 

29.0 

0.024 

64.9* 

21.4 

43.6 

0.007 

Character  education  (percent) 

66.9* 

26.3 

40.6 

0.001 

64.1 

43.5 

20.6 

0.129 

86.4* 

28.8 

57.6 

0.002 

Tolerance  and  diversity  (percent) 

37. 8A 

13.9 

23.9 

0.081 

38.6 

28.2 

10.4 

0.249 

46.2* 

19.2 

27.1 

0.006 

Risk  prevention  and  health  promotion 
(percent) 

29.4 

13.2 

16.2 

0.134 

43. 6A 

25.0 

18.6 

0.087 

37.0 

31.0 

6.0 

0.546 

Civic  responsibility  and  community  service 
(percent) 

13.5 

12.5 

1.1 

0.844 

26.9 

22.7 

4.2 

0.687 

26.5 

27.6 

-1.1 

0.930 

Any  SACD  goal  (percent) 

76.3* 

35.5 

40.7 

0.002 

84.1 

76.6 

7.6 

0.323 

80.3 

76.4 

3.9 

0.724 

Behavior  management  (percent) 

59.3 

68.5 

-9.2 

0.525 

72. 6A 

42.1 

30.6 

0.066 

58.4 

65.7 

-7.3 

0.609 

See  note  at  end  of  table. 
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Table  5.13.  Impacts  on  teacher-reported  SACD  classroom  activities — PA — Continued 


Panel  3:  Engagement  in  activities  to  promote  SACD  goals  linked  to  named  SACD  programs2 


Year  1 

Year  2 

Year  3 

(Spring  3rd  grade) 

(Spring  4th  grade) 

(Spring  5th  grade) 

SACD  activity 

T reat- 
ment 

Control 

Impact 

p-value 

T reat- 
ment 

Control 

Impact 

p-value 

T reat- 
ment 

Control 

Impact 

p-value 

Teacher  sample  size 

60 

54 

43 

39 

39 

38 

Violence  prevention  and  peace  promotion 
(percent) 

36.9 

23.4 

13.5 

0.152 

36.2 

22.5 

13.7 

0.215 

47.1 A 

16.7 

30.5 

0.066 

Social  and  emotional  development 
(percent) 

47.3* 

8.3 

39.0 

0.007 

48.4* 

7.3 

41.1 

0.001 

49.0* 

11.9 

37.1 

0.019 

Character  education  (percent) 

59.2* 

11.8 

47.4 

0.001 

t* 

t 

45.4 

0.001 

60.4* 

11.6 

48.9 

0.001 

Tolerance  and  diversity  (percent) 

31.2 

0.0 

31.2 

t 

t* 

t 

32.2 

0.014 

45.3* 

7.1 

38.2 

0.002 

Risk  prevention  and  health  promotion 
(percent) 

37.9 

19.4 

18.4 

0.110 

48.6* 

14.0 

34.6 

0.002 

44.6* 

18.4 

26.2 

0.030 

Civic  responsibility  and  community  service 
(percent) 

5.3 

0.0 

5.3 

T 

t* 

t 

19.1 

0.033 

23.0 

9.5 

13.5 

0.248 

Any  named  activity  (percent) 

77.3* 

30.7 

46.6 

0.002 

59.1 

46.2 

12.9 

0.260 

69.2 

42.2 

27.0 

0.121 

See  note  at  end  of  table. 
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Table  5.13.  Impacts  on  teacher-reported  SACD  classroom  activities — PA — Continued 

Panel  4:  Engagement  in  activities  to  promote  SACD  goals  linked  to  named  SACD  programs  for  at  least  1 hour  per  week 

Year  1 Year  2 Year  3 

(Spring  3rd  grade) (Spring  4th  grade) (Spring  5th  grade) 

T reat-  T reat-  T reat- 


SACD  activity 

ment 

Control 

Impact 

p-value 

ment 

Control 

Impact 

p-value 

ment 

Control 

Impact 

p-value 

Teacher  sample  size 

60 

54 

43 

39 

39 

38 

Violence  prevention  and  peace  promotion 
(percent) 

31. 9A 

14.1 

17.8 

0.051 

26.6 

12.5 

14.1 

0.211 

40.4* 

12.2 

28.2 

0.050 

Social  and  emotional  development 
(percent) 

40.3* 

4.8 

35.4 

0.027 

i* 

t 

36.9 

0.004 

43.7* 

10.0 

33.7 

0.014 

Character  education  (percent) 

49.2* 

5.3 

43.8 

0.002 

43.3 

0.0 

43.3 

t 

53.4* 

7.5 

46.0 

0.005 

Tolerance  and  diversity  (percent) 

28.0 

0.0 

28.0 

t 

27.4 

0.0 

27.4 

t 

39.0* 

7.6 

31.4 

0.002 

Risk  prevention  and  health  promotion 
(percent) 

26. 8A 

10.0 

16.8 

0.099 

37.4* 

7.8 

29.5 

0.031 

30. 2A 

9.6 

20.6 

0.060 

Civic  responsibility  and  community  service 
(percent) 

5.3 

0.0 

5.3 

t 

18.7 

0.0 

18.7 

t 

17.3 

7.9 

9.4 

0.441 

Any  named  activity  (percent) 

64.1* 

17.0 

47.1 

0.000 

52.4* 

16.5 

35.9 

0.018 

51.5 

30.6 

20.8 

0.242 

t Not  applicable. 

t Reporting  standards  not  met.  Values  suppressed  to  protect  confidentiality. 

* Treatment  group  significantly  different  from  control  group  at  the  .05  level. 

A Treatment  group  significantly  different  from  control  group  at  the  .10  to  > .05  level. 

1 1n  Year  1 , the  omnibus  impact  for  all  the  outcomes  measured  together  was  positive  and  statistically  significant  on  the  basis  of  a multivariate  statistical  test.  In  Years  1 and  3,  at  least 
one  outcome  remained  positive  and  statistically  significant  and  no  outcome  was  negative  and  statistically  significant  after  applying  the  Benjamini-Hochberg  (1995)  procedure  to  adjust 
significance  levels  downward  to  account  for  the  multiple  testing  of  impacts. 

2 In  Years  1 and  2,  at  least  half  of  the  impacts  were  positive  and  statistically  significant  and  no  impact  was  negative  and  statistically  significant  on  the  basis  of  univariate  statistical  tests. 
In  all  3 years,  at  least  one  outcome  remained  positive  and  statistically  significant  and  no  outcome  was  negative  and  statistically  significant  after  applying  the  Benjamini-Hochberg 
(1995)  procedure  to  adjust  significance  levels  downward  to  account  for  the  multiple  testing  of  impacts. 

NOTE:  Weights,  which  assign  equal  weight  to  each  school  within  the  program,  were  used  in  producing  the  treatment,  control,  and  overall  means. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Use  of  Materials  and  Strategies 

For  use  of  materials  and  strategies  to  support  SACD  goals,  87  comparisons  were  made,  with  4 expected  to  be 
significant  by  chance.  Eleven  significant  impacts  were  found  on  treatment  teachers’  use  of  materials  and 
strategies  in  all  3 years.  In  Year  1,  more  treatment  teachers  used  teacher  guides  (impact  = 32  percentage 
points),  student  materials  such  as  workbooks  or  worksheets  (impact  = 33  percentage  points),  giveaways 
(impact  = 37  percentage  points),  and  student-led/ student-assisted  instruction  (impact  =19  percentage  points) 
(table  5.14).  In  Year  2,  more  treatment  teachers  continued  to  use  teacher  guides  (impact  = 26  percentage 
points),  skill  training  (impact  = 28  percentage  points),  parent/ community  involvement  (impact  = 25 
percentage  points),  targeted  story  reading  or  writing  on  SACD  themes  (impact  = 19  percentage  points),  and 
time-out  for  negative  behavior  (impact  =16  percentage  points).  On  average,  treatment  teachers  also  used 
more  strategies  than  control  teachers  (by  1.5  strategies).  In  Year  3,  a significant  impact  was  seen  for  treatment 
teachers’  use  of  giveaways  (impact  = 23  percentage  points).  The  impact  on  the  domain  of  materials  and 
strategies  was  statistically  significant  in  Year  1. 
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Table  5.14.  Impacts  on  use  of  SACD  classroom  materials  and  teaching  strategies — PA 


Year  1 

(Spring  3rd  grade) 

Year  2 

(Spring  4th  grade) 

Year  3 

(Spring  5th  grade) 

SACD  material  and  teaching  strategy1 

T reat- 
ment 

Control 

Impact 

p-value 

T reat- 
ment 

Control  Impact 

p-value 

T reat- 
ment 

Control 

Impact 

p-value 

Teacher  sample  size 

60 

54 

43 

39 

39 

38 

Use  of  SACD  materials  (percent) 

Teacher  guides  (manuals,  curricula) 

86.7* 

54.5 

32.1 

0.017 

86.1* 

60.5 

25.6 

0.012 

90.5 

66.8 

23.7 

0.178 

Student  materials  (workbooks  or  sheets) 

84.8* 

51.8 

32.9 

0.007 

72.2 

58.3 

13.9 

0.243 

85.1 A 

66.8 

18.3 

0.076 

Instructional  aids  (games,  software,  videos) 

29.7 

34.5 

-4.7 

0.733 

38.4 

33.8 

4.6 

0.630 

45.1 

55.5 

-10.4 

0.340 

Giveaways  (bookmarks,  stickers) 

76.7* 

39.6 

37.1 

0.004 

68.7 

55.6 

13.1 

0.314 

79.0* 

56.3 

22.8 

0.019 

Children’s  literature 

33.9 

36.4 

-2.5 

0.823 

36.0 

49.4 

-13.5 

0.341 

34.2 

26.4 

7.8 

0.557 

Other  types  of  materials 

6.9 

15.2 

-8.4 

0.230 

19.7 

16.2 

3.5 

0.686 

t 

t 

-5.4 

0.265 

Did  not  use  any  of  these  materials 

tA 

t 

-20.5 

0.069 

t 

t 

-5.7 

0.299 

t 

t 

-3.3 

0.630 

Use  of  teaching  strategies  (percent) 

Role-playing 

80.1 

68.5 

11.6 

0.191 

80.1 

72.4 

7.7 

0.508 

83.2 

88.9 

-5.7 

0.542 

Cooperative  learning 

97.6 

96.3 

1.3 

0.706 

100.0 

97.1 

2.9 

t 

100.0 

100.0 

0.0 

t 

Peer  group  discussions 

93.1 

81.4 

11.7 

0.113 

90.9 

88.6 

2.3 

0.729 

97.0 

100.0 

-3.0 

t 

Direct  instruction  of  SACD 

86.9 

77.6 

9.2 

0.367 

94.4 

83.7 

10.8 

0.073 

100.0 

92.8 

7.2 

t 

Skill  training 

59.7 

46.7 

13.1 

0.187 

64.7* 

36.7 

28.0 

0.015 

81.1 

86.1 

-5.0 

0.563 

Incorporating  SACD  into  academic 
curriculum 

86.0 

75.8 

10.2 

0.149 

80.0 

68.7 

11.3 

0.260 

100.0 

97.9 

2.1 

t 

Parent  training 

t 

t 

-10.2 

0.118 

t 

t 

13.4 

0.116 

51. 3A 

27.0 

24.3 

0.077 

Parent/community  involvement 

7.8 

15.0 

-7.2 

0.322 

32.7* 

7.6 

25.1 

0.011 

60.1 

39.4 

20.7 

0.100 

Mentoring 

45.8 

37.0 

8.8 

0.445 

41.2 

41.2 

0.0 

1.000 

76.2 

51.4 

24.8 

0.137 

Good  behavior  notes  sent  home  daily 
or  weekly 

60.9 

51.8 

9.1 

0.390 

74.9 

59.8 

15.2 

0.157 

83.1 

87.9 

-4.8 

0.526 

Presenting  role  models 

68.9 

58.8 

10.1 

0.197 

65.4 

64.4 

1.0 

0.925 

85.5 

74.9 

10.6 

0.255 

See  notes  at  end  of  table. 
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Table  5.14.  Impacts  on  use  of  SACD  classroom  materials  and  teaching  strategies — PA — Continued 


Year  1 Year  2 Year  3 

(Spring  3rd  grade) (Spring  4th  grade) (Spring  5th  grade) 

Treat-  Treat-  Treat- 

SACD  material  and  teaching  strategy1 ment  Control  Impact  p-value ment  Control  Impact  p-value ment  Control  Impact  p-value 

Use  of  teaching  strategies  (percent) — 

Continued 

Targeted  story  reading  or  writing  on  social 


and  character  development  themes 

94. 8A 

80.9 

13.9 

0.083 

98.5* 

79.6 

18.9 

0.015 

97.6 

97.9 

-0.2 

0.939 

Peer  mediation 

59.7 

56.0 

3.8 

0.619 

61.3 

62.1 

-0.9 

0.917 

92.0 

80.2 

11.9 

0.174 

Honor  roll  for  positive  behavior 

67.3 

69.1 

-1.7 

0.854 

58.6 

75.4 

-16.7 

0.108 

81.5 

89.5 

-8.0 

0.375 

Pledges  or  recitations  on  social  and 
character  development  themes 

48.9 

54.0 

-5.0 

0.664 

57.2 

55.9 

1.4 

0.934 

75.6 

72.0 

3.7 

0.774 

Guided  visualization 

62.5 

53.4 

9.0 

0.492 

69.6 

60.7 

9.0 

0.485 

77.1 

76.8 

0.3 

0.977 

Student-led/student-assisted  instruction 

65.1* 

45.9 

19.2 

0.039 

59.6 

48.8 

10.8 

0.265 

90.6 

79.1 

11.5 

0.115 

Journaling 

81.8 

80.3 

1.5 

0.880 

77.6 

87.8 

-10.2 

0.411 

95.2 

97.6 

-2.3 

0.631 

Time  out  for  negative  behavior 

84.3 

88.9 

-4.6 

0.562 

93.8* 

77.6 

16.2 

0.031 

100.0 

97.6 

2.4 

t 

Daily  or  weekly  rewards  for  positive 
behavior 

91.8 

89.9 

1.9 

0.696 

97.6 

92.7 

4.9 

0.344 

98.0 

97.9 

0.1 

0.986 

Any  strategy 

100.0 

100.0 

0.0 

t 

100.0 

100.0 

0.0 

T 

100.0 

100.0 

0.0 

t 

Number  of  strategies  (mean) 

13.3 

12.4 

1.0 

0.202 

14.0* 

12.5 

1.5 

0.042 

16.9 

16.0 

0.9 

0.170 

t Not  applicable. 

$ Reporting  standards  not  met.  Values  suppressed  to  protect  confidentiality. 

* Treatment  group  significantly  different  from  control  group  at  the  .05  level. 

A Treatment  group  significantly  different  from  control  group  at  the  .10  to  > .05  level. 

1 1n  Year  1 , at  least  one  outcome  remained  positive  and  statistically  significant  and  no  outcome  was  negative  and  statistically  significant  after  applying  the  Benjamini-Hochberg  (1995) 
procedure  to  adjust  significance  levels  downward  to  account  for  the  multiple  testing  of  impacts. 

NOTE:  Weights,  which  assign  equal  weight  to  each  school  within  the  program,  were  used  in  producing  the  treatment,  control,  and  overall  means. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Regarding  the  use  of  schoolwide  strategies,  18  comparisons  were  made  between  treatment  and  control 
teacher  reports,  with  1 expected  to  be  significant  by  chance.  There  were  2 statistically  significant  differences 
between  treatment  and  control  teachers’  reported  use  of  schoolwide  strategies  (these  data  are  not  shown  in  a 
table),  with  treatment  teachers  reporting  more  use  of  unspecified  schoolwide  activities  than  control  teachers 
(impact  =13  percentage  points)  in  Year  2,  and  more  use  of  morning  announcements  or  videos  (impact  = 36 
percentage  points)  in  Year  3.  The  overall  impact  of  the  PA  program  on  the  domain  for  use  of  schoolwide 
strategies  in  named  SACD  activities  was  not  significant  in  any  year. 

Participation  in  Professional  Development 

Regarding  reported  participation  in  professional  development,  27  comparisons  were  made  over  3 years,  with 
1 expected  to  be  significant  by  chance.  The  intervention  had  a statistically  significant  effect  on  treatment 
teachers’  participation  in  professional  development,  with  more  treatment  teachers  than  control  teachers 
reporting  SACD  training  in  the  past  12  months  (impact  = 35  percentage  points)  and  more  hours  of  training 
(by  4.9  hours  on  average)  in  Year  1.  In  Years  1 and  2,  there  were  significant  impacts  on  training  for  character 
education  (impact  = 29  and  28  percentage  points).  No  significant  impacts  were  found  on  the  domain  in  any 
of  the  3 years. 
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Table  5.15.  Impacts  on  teacher-reported  SACD  professional  development — PA 


Year  1 Year  2 Year  3 

(Spring  3rd  grade) (Spring  4th  grade) (Spring  5th  grade) 

T reat-  T reat-  T reat- 


SACD  professional  development 

ment 

Control 

Impact 

p-value 

ment 

Control 

Impact 

p-value 

ment 

Control 

Impact 

p-value 

Teacher  sample  size 

60 

54 

43 

39 

39 

38 

SACD  training  in  past  12  months  (percent) 

89.3* 

54.1 

35.3 

0.015 

73.0 

68.5 

4.5 

0.623 

62.9 

58.0 

5.0 

0.682 

Hours  of  SACD  training  (mean) 

8.0* 

3.1 

4.9 

0.011 

4.2 

4.3 

-0.1 

0.944 

4.0 

4.5 

-0.5 

0.737 

Training  by  goal  (percent) 

Violence  prevention  and  peace  promotion 

21.2 

13.1 

8.1 

0.447 

18.5 

24.7 

-6.2 

0.528 

13.0 

8.8 

4.2 

0.543 

Social  and  emotional  development 

31.4 

21.2 

10.2 

0.434 

24.6 

30.4 

-5.8 

0.654 

30.7 

13.6 

17.1 

0.122 

Character  education 

63.1* 

34.2 

29.0 

0.040 

45.8* 

18.2 

27.6 

0.025 

35. 5A 

13.6 

21.9 

0.086 

Tolerance  and  diversity 

29.1 

10.3 

18.8 

0.125 

16.6 

16.4 

0.2 

0.984 

11.8 

25.0 

-13.2 

0.208 

Risk  prevention  and  health  promotion 

11.2 

9.0 

2.1 

0.778 

9.6 

15.2 

-5.6 

0.491 

20.2 

17.7 

2.5 

0.822 

Civic  responsibility  and  community  service 

t 

t 

8.2 

0.179 

7.3 

7.5 

-0.2 

0.974 

0.0 

6.8 

-6.8 

t 

Behavior  management 

42.0 

24.8 

17.2 

0.238 

37.0 

46.3 

-9.3 

0.447 

38.4 

24.8 

13.5 

0.369 

t Not  applicable. 

$ Reporting  standards  not  met.  Values  suppressed  to  protect  confidentiality. 

* Treatment  group  significantly  different  from  control  group  at  the  .05  level. 

A T reatment  group  significantly  different  from  control  group  at  the  . 1 0 to  > .05  level. 

NOTE:  Weights,  which  assign  equal  weight  to  each  school  within  the  program,  were  used  in  producing  the  treatment,  control,  and  overall  means. 
SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Attitudes  and  Practices 

Teachers  reported  on  their  enthusiasm  for  SACD  efforts  in  their  schools  (these  data  are  not  shown  in  a table) 
by  indicating  enthusiasm,  cooperation,  or  open  dislike.  They  also  reported  on  the  SACD  practices  of  teachers 
and  staff  members  in  their  schools  (these  data  are  not  shown  in  a table).  These  practices  included  modeling 
positive  character  and  behavior  traits  with  students  and  fellow  teachers,  involving  students  in  making 
decisions,  giving  students  a voice  in  school  governance,  the  school  encouraging  parent  involvement  in 
children’s  social  and  character  development,  and  using  developmentally  appropriate  discipline  strategies 
rather  than  punishment  for  misbehavior.  Twenty-seven  comparisons  were  made  over  3 years,  with  1 expected 
to  be  significant  by  chance.  In  Year  1,  there  was  a statistically  significant  impact  on  teachers’  enthusiasm  for 
SACD  efforts  in  their  schools  (out  of  9 comparisons  made),  with  more  treatment  teachers  reporting 
enthusiasm  than  control  teachers  (impact  = 4 percentage  points).  There  were  no  significant  differences  in 
treatment  and  control  teacher  reports  of  the  overall  use  of  practices  conducive  to  students’  social  and 
character  development  for  any  of  the  years.  There  was  a significant  positive  impact  on  the  domain  in  Year  1. 

Year-by-Year  Impacts  on  Students  and  Perceptions  of  School  Climate 

The  primary  research  question  for  the  PA  evaluation  was  this: 

What  is  the  effect  of  the  PA  program  on  children’s  social  and  emotional  competence, 
behavior,  academics,  and  perceptions  of  school  climate? 

The  first  approach  to  answering  this  question  was  to  examine  the  year-by-year  impacts  of  the  PA  program  on 
these  student  and  school  climate  outcomes  over  the  3 years  as  the  students  progressed  from  third  through 
fifth  grades. 

Equation  (2)  (described  in  chapter  1)  was  estimated  to  provide  the  impacts  of  the  PA  program  on  the  20 
outcomes  using  data  from  the  14  treatment  and  control  schools.  For  the  PA  evaluation,  equation  (2)  excluded 
the  program  fixed  effects  (6^)  and  included  program-specific  covariates  and  random  school  effects  covariates. 
Table  5.16  lists  the  covariates  used  with  outcomes  from  each  report  in  the  PA  analysis. 
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Table  5.16.  Covariates  used  with  outcomes  from  each  report  for  analysis — PA 


Potential  covariate 

CR 

outcome 

PCR 

outcome 

TRS 

outcome 

TRCS 

outcome 

Total  number 

20 

26 

22 

7 

Child-reported 

Female 

Y 

Y 

Y 

Hispanic 

Y 

Y 

Y 

Black  (non-Hispanic) 

Y 

Y 

Y 

Other  ethnicity 

Y 

Y 

Y 

Age  in  years 

Y 

Y 

Y 

Scales 

Afraid  at  School 

Altruistic  Behavior 

Y 

Empathy 

Engagement  with  Learning 

Y 

Negative  School  Orientation 

Y 

Normative  Beliefs  About  Aggression 

Y 

Y 

Y 

Sense  of  School  as  a Community 

Problem  Behavior 

Y 

Y 

Y 

Self-Efficacy  for  Peer  Interactions 

Y 

Victimization  at  School 

Y 

Y 

Primary  caregiver-reported 

Age  in  years 

Y 

Y 

Completed  high  school  or  eguivalent 

Y 

Y 

Y 

Some  college 

Y 

Y 

Y 

Bachelor’s  or  higher  degree 

Y 

Y 

Y 

Highest  level  of  education  in  household 

Completed  high  school  or  eguivalent 

Y 

Some  college 

>4 

Bachelor’s  or  higher  degree 

V 

Mother  present  in  home  life 

Mother  and  father  present 

Y 

Y 

Respondent  someone  other  than  mother  or  father 

Y 

Y 

Number  of  people  in  household 

V 

Y 

Y 

Household  income:  $20,000  to  $40,000 

Household  income:  $40,000  to  $60,000 

Household  income:  More  than  $60,000 

Income-to-poverty-threshold  ratio:  Below  135  percent 

V 

Y 

Income-to-poverty-threshold  ratio:  135  to  185  percent 

V 

Y 

Full-time  employment 

Y 

Y 

Part-time  employment 

Y 

Y 

See  notes  at  end  of  table. 
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Table  5.16.  Covariates  used  with  outcomes  from  each  report  for  analysis — PA — Continued 


Potential  covariate 

CR 

outcome 

PCR 

outcome 

TRS 

outcome 

TRCS 

outcome 

Parental  scales 

APQ-Poor  Monitoring  and  Supervision  Subscale 

V 

APQ-Positive  Parenting  Subscale 

Child-Centered  Social  Control 

Confusion,  Hubbub,  and  Order 

S 

•/ 

S 

Community  Resources 

Community  Risk 

•/ 

Parent  and  Teacher  Involvement 

Child  scales 

Altruistic  Behavior 

Positive  Social  Behavior 

S 

Problem  Behavior 

Teacher-reported 

Female 

V 

Hispanic 

•/ 

Black  (non-Hispanic) 

•/ 

Other  ethnicity 

•/ 

Total  teaching  experience 

Total  experience  in  current  school 

Regular  certificate 

•/ 

Other  certificate 

•/ 

Highest  degree-bachelor’s 

Child  scales 

Academic  Competence  and  Motivation 

ADHD-Related  Behavior 

s 

Altruistic  Behavior 

•/ 

s 

Positive  Social  Behavior 

s 

Problem  Behavior 

•/ 

Parent  and  Teacher  Involvement 

NOTE:  Abbreviations  are 
CR:  Child  Report 
PCR:  Primary  Caregiver  Report 
TRS:  Teacher  Report  on  Student 
TRCS:  Teacher  Report  on  Classroom  and  School 
ADHD:  Attention  deficit  hyperactivity  disorder 
APQ:  Alabama  Parenting  Questionnaire 
S : Covariate  used 
Blank  cell:  Covariate  not  used 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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To  assess  the  statistical  power  of  the  program -level  impact  estimates,  minimum  detectable  impacts  in  effect 
size  units  (MDES)  for  each  outcome  measure  were  calculated  for  the  PA  evaluation  (table  5.17).  MDES 
represent  the  smallest  impacts  in  effect  size  (standard  deviation)  units  that  can  be  detected  with  a high 
probability  (80%).  MDES  are  primarily  a function  of  study  sample  sizes,  the  degrees  of  freedom  available  for 
statistical  tests,  and  design  effects  due  to  clustering  (Schochet  2005).  For  the  PA  evaluation,  the  MDES  range 
from  0.096  to  0.953  for  the  child-level  outcomes  based  on  the  Child,  Caregiver,  and  Teacher  Report  on 
Student  and  from  0.321  to  0.742  for  the  school  climate  outcomes  based  on  the  Teacher  Report  on  Classroom 
and  School.  In  general,  the  MDES  for  the  school  climate  outcomes  were  larger  than  those  for  the  child-level 
outcomes. 
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Table  5.17.  Adjusted  minimum  detectable  effect  sizes  for  impact  evaluation — PA 


Outcome  measure-Report 

Year  1 

Year  2 

Year  3 

Social  and  Emotional  Competence  Domain 

Self-Efficacy  for  Peer  Interaction-CR 

0.203 

0.141 

0.168 

Normative  Beliefs  About  Aggression-CR 

0.241 

0.350 

0.203 

Empathy-CR 

0.256 

0.296 

0.178 

Behavior  Domain 

Altruistic  Behavior-CR 

0.096 

0.278 

0.103 

Altruistic  Behavior-PCR 

0.132 

0.162 

0.162 

Altruistic  Behavior-TRS 

0.631 

0.953 

0.784 

Positive  Social  Behavior-PCR 

0.106 

0.165 

0.199 

Positive  Social  Behavior-TRS 

0.310 

0.392 

0.285 

Problem  Behavior-CR 

0.231 

0.291 

0.233 

Problem  Behavior-PCR 

0.159 

0.150 

0.130 

Problem  Behavior-TRS 

0.268 

0.206 

0.244 

ADHD-Related  Behavior-TRS 

0.204 

0.198 

0.285 

Academics  Domain 

Engagement  with  Learning-CR 

0.096 

0.154 

0.223 

Academic  Competence  and  Motivation-TRS 

0.145 

0.174 

0.263 

Perceptions  of  School  Climate  Domain 

Positive  School  Orientation-CR 

0.333 

0.350 

0.373 

Negative  School  Orientation-CR 

0.096 

0.259 

0.335 

Student  Afraid  at  School-CR 

0.117 

0.183 

0.186 

Victimization  at  School-CR 

0.097 

0.393 

0.175 

Feelings  of  Safety-TRCS 

0.489 

0.384 

0.742 

Student  Support  for  Teachers-TRCS 

0.468 

0.321 

0.602 

NOTE:  Abbreviations  are 
CR:  Child  Report 
PCR:  Primary  Caregiver  Report 
TRS:  Teacher  Report  on  Student 
TRCS:  Teacher  Report  on  Classroom  and  School 
ADHD:  Attention  deficit  hyperactivity  disorder 
The  minimum  detectable  effect  (MDE)  formula  used  in  the  calculations  is  as  follows: 


MDE  = factor(df)  * 


Pi  (-  + -)  + (l-Pi)(— + — ) 
\sT  scJ  \sTnT  scnc) 


where  Sj-and  sc  are  the  number  of  treatment  and  comparison  schools;  nT  and  nc  are  the  average  number  of  students  per  classroom; 
pi  is  the  intraclass  correlation  (ICC)  at  the  school  level;  and  factor(df)  is  a constant  that  depends  on  the  number  of  degrees  of 
freedom  (df)  available  for  analysis  (and  is  2.802  for  the  pooled  analysis). 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Table  5.18  provides  the  estimates  of  the  PA  program’s  impacts  on  each  of  the  20  outcomes  over  each  of  the 
3 years  (60  impacts  in  total,  with  3 expected  to  be  statistically  significant  by  chance).  Of  the  60  results,  3 were 
statistically  significant  (2  beneficial  and  1 detrimental).  In  Year  1,  the  PA  program  had  a detrimental  impact 
on  Engagement  with  Learning  (Child  Report,  effect  size  [ES]  = -0.25)  and  a substantively  important  but 
nonsignificant  beneficial  impact  on  Altruistic  Behavior  (Teacher  Report  on  Student,  ES  = 0.27).  In  Year  2, 
beneficial  impacts  were  found  for  Positive  Social  Behavior  (Primary  Caregiver  Report,  ES  = 0.24)  and 
Problem  Behavior  (Teacher  Report  on  Student,  ES  = -0.24).  Substantively  important  but  nonsignificant 
beneficial  impact  was  found  in  Years  2 and  3 on  Student  Support  for  Teachers  (Teacher  Report  on 
Classroom  and  School,  ES  = 0.28  and  0.27).  Application  of  the  heuristic  to  adjust  for  multiple  comparisons 
within  each  outcome  domain  indicates  that  PA  had  a statistically  significant  detrimental  impact  on  Academics 
in  Year  1. 
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Table  5.18.  Impacts  on  child  and  school  outcomes — PA 


Year  1 Year  2 Year  3 

(Spring  3rd  grade) (Spring  4th  grade) (Spring  5th  grade) 


Treat-  Effect  Treat-  Effect  Treat-  Effect 


Scale-Report 

ment 

Control 

size 

p-value 

ment 

Control 

size 

p-value 

ment 

Control 

size 

p-value 

Social  and  Emotional  Competence  Domain 

Self-Efficacy  for  Peer  Interactions-CR  (+) 

2.99A 

3.13 

-0.24 

0.090 

3.15 

3.22 

-0.12 

0.258 

3.20 

3.18 

0.03 

0.823 

Normative  Beliefs  About  Aggression-CR  (-) 

1.34 

1.42 

-0.13 

0.388 

1.48 

1.65 

-0.21 

0.303 

1.50 

1.60 

-0.16 

0.264 

Empathy-CR  (+) 

2.28 

2.18 

0.21 

0.176 

2.14 

2.07 

0.14 

0.422 

2.05 

1.99 

0.16 

0.255 

Behavior  Domain 

Altruistic  Behavior-CR  (+) 

1.37 

1.45 

-0.09 

0.289 

1.25 

1.25 

0.00 

0.998 

1.25 

1.16 

0.14 

0.198 

Altruistic  Behavior-PCR  (+) 

2.42 

2.42 

0.01 

0.927 

2.49 

2.34 

0.19 

0.139 

2.49 

2.38 

0.16 

0.281 

Altruistic  Behavior-TRS  (+) 

1.59 

1.45 

0.27s 

0.480 

1.62 

1.50 

0.21 

0.728 

1.40 

1.38 

0.03 

0.958 

Positive  Social  Behavior-PCR  (+) 

2.99 

2.95 

0.07 

0.342 

3.03* 

2.89 

0.24 

0.039 

3.03 

2.96 

0.14 

0.360 

Positive  Social  Behavior-TRS  (+) 

2.95 

2.92 

0.04 

0.740 

3.00 

2.86 

0.19 

0.365 

2.98 

2.83 

0.20 

0.238 

Problem  Behavior-CR  (-) 

0.51 

0.47 

0.07 

0.632 

0.59 

0.58 

0.01 

0.948 

0.67 

0.74 

-0.10 

0.494 

Problem  Behavior-PCR  (-) 

1.55 

1.60 

-0.10 

0.261 

1.57 

1.53 

0.11 

0.320 

1.52 

1.55 

-0.08 

0.485 

Problem  Behavior-TRS  (-) 

1.53 

1.58 

-0.08 

0.486 

1.48* 

1.61 

-0.24 

0.048 

1.51 

1.62 

-0.16 

0.222 

ADHD-Related  Behavior-TRS  (-) 

1.72 

1.77 

-0.08 

0.415 

1.69 

1.75 

-0.11 

0.357 

1.71 

1.78 

-0.10 

0.533 

Academics  Domain1 

Engagement  with  Learning-CR  (+) 

3.56* 

3.69 

-0.25 

0.017 

3.58 

3.64 

-0.11 

0.390 

3.56 

3.54 

0.03 

0.862 

Academic  Competence  and  Motivation- 

TRS  (+) 

2.78 

2.80 

-0.02 

0.783 

2.80 

2.79 

0.01 

0.933 

2.67 

2.70 

-0.03 

0.818 

See  notes  at  end  of  table. 
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Table  5.18.  Impacts  on  child  and  school  outcomes — PA — Continued 


Year  1 Year  2 Year  3 

(Spring  3rd  grade) (Spring  4th  grade) (Spring  5th  grade) 


Treat-  Effect  Treat-  Effect  Treat-  Effect 


Scale-Report 

ment 

Control 

size 

p-value 

ment 

Control 

size 

p-value 

ment 

Control 

size 

p-value 

Perceptions  of  School  Climate  Domain 
Positive  School  Orientation-CR  (+) 

2.73 

2.65 

0.11 

0.576 

2.44 

2.47 

-0.04 

0.833 

2.41 

2.32 

0.14 

0.565 

Negative  School  Orientation-CR  (-) 

2.14 

2.15 

-0.02 

0.837 

2.17 

2.22 

-0.08 

0.628 

2.24 

2.30 

-0.12 

0.558 

Student  Afraid  at  School-CR  (-) 

2.53 

2.55 

-0.02 

0.788 

2.41 

2.42 

-0.02 

0.892 

2.35 

2.39 

-0.05 

0.716 

Victimization  at  School-CR  (-) 

0.90 

0.82 

0.10 

0.228 

0.77 

0.67 

0.13 

0.578 

0.95 

0.83 

0.16 

0.244 

Feelings  of  Safety-TRCS  (+) 

3.25 

3.09 

0.15 

0.473 

3.19 

3.31 

-0.13 

0.494 

3.07 

3.23 

-0.18 

0.618 

Student  Support  for  Teachers-TRCS  (+) 

3.05 

2.92 

0.13 

0.418 

3.27 

2.99 

0.28° 

0.113 

3.12 

2.91 

0.27s 

0.419 

* Treatment  group  significantly  different  from  control  group  at  the  .05  level. 

A Treatment  group  significantly  different  from  control  group  at  the  .10  to  > .05  level. 
a Substantive  (but  nonsignificant  at  .05  level)  effect  size  of  2 .25  or  < -.25. 

1 1n  Year  1 , at  least  half  of  the  impacts  were  positive  and  statistically  significant  and  no  impact  was  negative  and  statistically  significant  based  on  univariate  statistical  tests,  and  at  least 
one  outcome  remained  positive  and  statistically  significant  and  no  outcome  was  negative  and  statistically  significant  after  applying  the  Benjamini-Hochberg  (1995)  procedure  to  adjust 
significance  levels  downward  to  account  for  the  multiple  testing  of  impacts.  The  number  of  results  found  significant  was  no  more  than  expected  by  chance. 

NOTE:  Abbreviations  are 
CR:  Child  Report 
PCR:  Primary  Caregiver  Report 
TRS:  Teacher  Report  on  Student 
TRCS:  Teacher  Report  on  Classroom  and  School 
ADHD:  Attention  deficit  hyperactivity  disorder 

The  +/-  signs  in  parentheses  indicate  the  direction  of  a beneficial  outcome.  All  impact  estimates  were  calculated  using  regression  models  where  each  program  and  school  within  a 
program  was  weighted  equally.  The  standard  errors  of  all  estimates  account  for  design  effects  due  to  unequal  weighting  and  the  clustering  of  students  within  schools.  See  table  1 .5  for 
information  about  the  measures  used  to  create  the  outcome  variables.  The  effect  size  was  calculated  by  dividing  the  estimated  impact  by  the  standard  deviation  of  the  outcome 
measure  for  the  control  group. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Impacts  on  Child  Outcomes  Over  Time 

PA  impacts  on  the  child  outcomes  over  time  were  estimated  using  growth  curve  models  by  examining 
treatment  and  control  group  differences  in  the  trajectories  of  student  outcomes  during  the  follow-up  period 
while  accounting  for  clustering  at  the  school  level.  The  growth  curve  models  are  estimated  using  a three-level 
hierarchical  linear  model,  where  Level  1 corresponds  to  time,  Level  2 to  students,  and  Level  3 to  schools 
(described  in  chapter  1). 

Table  5.19  provides  the  estimates  of  the  PA  program’s  impacts  on  the  growth  in  student  outcomes  over  the  3 
years.  The  estimated  impacts  ranged  in  effect  size  units  (absolute  value)  from  0.00  to  0.09.  None  of  the  18 
estimated  PA  intervention  impacts  on  the  trajectories  of  child  outcomes  was  statistically  significant  (1  in  18  is 
expected  to  be  significant  by  chance). 
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Table  5.19.  Impacts  on  growth  of  child  outcomes — PA 


Average  growth  in  the  score  per  year 

1 

Scale-Report 

Mean  score  at 
implementation2 

Treatment 

Control 

Impact  on 
growth3 

Standard 
Effect  error  of 

size4  impact 

p-value  of 
impact 

Social  and  Emotional  Competence  Domain 

Self-Efficacy  for  Peer  Interactions-CR  (+) 

2.92 

0.14 

0.13 

0.01 

0.02 

0.03 

0.756 

Normative  Beliefs  About  Aggression-CR  (-) 

1.28 

0.14 

0.13 

0.01 

0.01 

0.05 

0.911 

Empathy-CR  (+) 

2.37 

-0.15 

-0.15 

0.00 

0.01 

0.02 

0.925 

Behavior  Domain 

Altruistic  Behavior-CR  (+) 

1.59 

-0.14 

-0.19 

0.05 

0.06 

0.03 

0.150 

Altruistic  Behavior-PCR  (+) 

2.43 

0.01 

-0.07 

0.08 

0.09 

0.05 

0.100 

Altruistic  Behavior-TRS  (+) 

1.45 

0.00 

0.00 

0.00 

-0.01 

0.09 

0.965 

Positive  Social  Behavior-PCR  (+) 

2.92 

0.03 

-0.01 

0.04 

0.07 

0.03 

0.119 

Positive  Social  Behavior-TRS  (+) 

2.85 

0.02 

-0.05 

0.07 

0.08 

0.06 

0.310 

Problem  Behavior-CR  (-) 

0.33 

0.17 

0.13 

0.04 

0.08 

0.05 

0.506 

Problem  Behavior-PCR  (-) 

1.59 

-0.01 

0.01 

-0.01 

-0.03 

0.02 

0.392 

Problem  Behavior-TRS  (-) 

1.48 

0.05 

0.09 

-0.04 

-0.06 

0.03 

0.245 

ADHD-Related  Behavior-TRS  (-) 

1.77 

0.01 

0.03 

-0.01 

-0.01 

0.05 

0.796 

Academics  Domain 

Engagement  with  Learning-CR  (+) 

3.60 

0.00 

0.01 

-0.01 

-0.01 

0.04 

0.834 

Academic  Competence  and  Motivation-TRS  (+) 

2.71 

-0.05 

-0.02 

-0.03 

-0.02 

0.05 

0.609 

See  notes  at  end  of  table. 
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Table  5.19.  Impacts  on  growth  of  child  outcomes — PA — Continued 


Average  growth  in  the  score  per  year1 


Scale-Report 

Mean  score  at 
implementation2 

Treatment 

Control 

Impact  on 
growth3 

Effect 

4 

size 

Standard 
error  of 
impact 

p-value  of 
impact 

Perceptions  of  School  Climate  Domain 
Positive  School  Orientation-CR  (+) 

2.93 

-0.24 

-0.24 

0.00 

0.00 

0.05 

0.977 

Negative  School  Orientation-CR  (-) 

2.10 

0.05 

0.08 

-0.03 

-0.04 

0.03 

0.338 

Student  Afraid  at  School-CR  (-) 

2.59 

-0.05 

-0.13 

0.07 

0.09 

0.05 

0.156 

Victimization  at  School-CR  (-) 

0.87 

-0.04 

-0.02 

-0.02 

-0.02 

0.05 

0.639 

1 Pertains  to  the  estimated  slope  of  the  outcome  for  the  treatment  or  control  groups. 

2 The  average  score  at  implementation  is  calculated  across  treatment  and  control  groups,  using  regression  models  for  adjustment  on  covariates. 

3 Estimated  difference  between  the  slope  of  the  treatment  and  control  groups. 

4 Effect  size:  the  slope  of  the  treatment  group  minus  the  slope  of  the  control  group  divided  by  the  standard  deviation  of  the  outcome  for  the  program's  control  group  (the  standard 
deviation  is  calculated  without  accounting  for  school-level  clustering  or  regression  adjustments). 

NOTE:  Abbreviations  are 
CR:  Child  Report 
PCR:  Primary  Caregiver  Report 
TRS:  Teacher  Report  on  Student 
ADHD:  Attention  deficit  hyperactivity  disorder 

No  findings  were  found  statistically  significant  at  or  below  the  .05  level.  The  +/-  signs  in  parentheses  indicate  the  direction  of  a beneficial  outcome.  All  impact  estimates  were  calculated 
using  HLM  6.06.  Sample  weights  were  used  in  all  analyses  to  give  (1)  each  school  equal  weight  in  each  program  (within  each  time  period)  and  (2)  each  time  period  equal  weight  within 
the  analysis.  See  table  1 .5  for  information  about  the  measures  used  to  create  the  outcome  variables. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Summary 

As  part  of  the  Social  and  Character  Development  (SACD)  Research  Program,  researchers  at  the  Illinois  site 
implemented  and  evaluated  the  PA  program.  This  program  focused  on  teaching  children  about  the  benefits 
of  physical,  intellectual,  social,  and  emotional  positive  actions  with  a curriculum  designed  to  enhance 
children’s  self-concept  and  self-regulatory  skills  through  the  identification  of  positive  thoughts,  feelings,  and 
actions.  Fourteen  public  schools  in  a single  large,  urban  school  district  in  Illinois  were  recruited  by  the  Illinois 
research  team  and  randomly  assigned  to  treatment  and  control  conditions  to  determine  the  impact  of  the  PA 
program  on  social  and  character  development  activities  in  the  schools  and  on  the  child  outcome  domains  of 
Social  and  Emotional  Competence,  Behavior,  Academics,  and  Perceptions  of  School  Climate. 

Analyses  of  the  initial  characteristics  of  the  sample  (students,  caregivers,  communities,  teachers,  and  schools) 
indicated  that  randomization  to  treatment  and  control  status  produced  groups  that  were  similar  on  the 
observed  characteristics  at  the  start  of  the  study.  Documentation  of  the  initial  level  of  social  and  character 
development  activities  in  the  schools  revealed  few  differences  (3,  the  number  that  would  be  expected  by 
chance  out  of  62  comparisons)  between  the  treatment  and  control  teachers  and  classrooms.  This  is  important 
for  two  reasons:  one,  it  indicates  that  randomization  created  comparable  groups  near  the  start  of  the  study, 
and  two,  it  shows  that  treatment  and  control  schools  both  had  high  levels  of  SACD  activities  near  the  start  of 
the  study,  indicating  that  social  and  character  development  activities  are  part  of  the  “standard  practice”  of 
these  schools  in  Illinois.  Standard  practice  at  all  of  the  schools  included  reports  of  45  percent  to  81  percent  of 
teachers  using  SACD  activities,  92  percent  of  teachers  using  specific  materials  in  conjunction  with  these 
activities,  99  percent  of  teachers  using  at  least  one  of  the  specified  instructional  strategies,  and  68  percent  of 
teachers  participating  in  SACD  training  over  the  past  12  months. 

Analyses  of  the  impacts  of  the  PA  program  on  the  use  of  SACD  activities  in  the  schools  revealed  impacts  on 
the  use  of  such  activities  (33  out  of  90)  and  related  materials  and  strategies  (1 3 out  of  87)  across  the  3 years, 
and  use  of  more  professional  development  activities  for  treatment  teachers  in  all  years  (4  out  of  27). 

Of  the  20  child-level  outcomes  representing  the  four  domains  of  Social  and  Emotional  Competence, 
Behavior,  Academics,  and  Perceptions  of  School  Climate  assessed  in  each  of  the  3 years  of  the  study  (a  total 
of  60  results),  3 were  statistically  significant  (2  beneficial  and  1 detrimental).  A growth  curve  analysis  was  used 
to  analyze  the  change  over  time  in  these  same  outcomes  between  initial  data  collection  and  the  final  outcome 
data  collection  at  the  end  of  the  study.  None  of  the  18  child -level  outcomes  assessed  showed  that  PA  had  a 
significant  impact. 

The  SACD  evaluation  did  not  find  evidence  to  support  the  hypothesis  that  PA  had  beneficial  impacts  on 
students’  social  and  character  development.  Such  results  could  be  caused  by  the  inability  of  the  program  to 
cause  such  change,  possibly  because  the  theory  of  action  for  the  program  is  incomplete  or  the  activities  to 
carry  out  that  theory  are  not  effective. 

However,  these  results  may  also  be  due  to  the  inability  of  the  evaluation  to  observe  such  a change  due  to  the 
control  condition,  the  level  of  nonparticipation,  or  the  sample  size.  The  control  schools  continued  using  their 
standard  SACD  activities,  and  these  turned  out  to  be  high  in  quantity  and  broad  in  scope.  In  2004  Illinois 
passed  the  Illinois  Children’s  Mental  Health  Act  (Public  Act  93-0495),  Section  15(a),  which  calls  on  the 
Illinois  State  Board  of  Education  to  “develop  and  implement  a plan  to  incorporate  social  and  emotional 
development  standards  as  part  of  the  Illinois  Learning  Standards.”  This  state -level  change  may  have  led 
control  schools  to  implement  increasing  amounts  of  SACD  activities  during  the  course  of  this  study.  While 
the  PA  program  had  a significant  positive  impact  on  the  amount  and  type  of  SACD  activities,  the  resulting 
difference  between  the  treatment  and  control  schools  in  the  amount  of  SACD  activities  may  not  have  been 
large  enough  to  cause  significant  differences  in  the  student  outcomes. 
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Second,  about  one-quarter  to  one-third  of  the  students  in  the  sample  universe  did  not  take  part  (depending 
on  year)  because  of  nonconsent  or  noncompletion  of  the  surveys.  As  a determination  could  not  be  made  as 
to  whether  the  students  not  taking  part  significantly  differed  from  those  who  did  take  part,  the  evaluation’s 
results  are  valid  only  for  the  students  who  took  part.  If  the  students  not  taking  part  were  different,  and  if  they 
would  have  responded  better  to  PA  than  to  the  SACD  activities  occurring  in  the  control  schools,  then  the 
evaluation  could  have  underestimated  the  program’s  impact.  Third,  the  sample  size  of  14  schools  and  the 
resulting  higher  MDES  compared  to  those  for  the  multiprogram  evaluation  may  have  reduced  the  likelihood 
of  detecting  statistically  significant  effects.  However,  it  should  be  noted  that  57  percent  of  the  MDES  for  the 
60  outcomes  used  in  the  year-by-year  analysis  were  below  0.25  (42%  were  below  0.20). 
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Chapter  6.  Promoting  Alternative  Thinking  Strategies 

(PATHS) 


The  Children’s  Institute 
(New  York/Minnesota  site) 


Intervention 

Researchers  at  the  Children’s  Institute  in  Rochester  (New  York/Minnesota  site)  evaluated  the  Promoting 
Alternative  Thinking  Strategies  (PATHS)  program  (Kusche  and  Greenberg  1994).  PATHS  is  designed  to 
enhance  social  and  emotional  competence  and  understanding  in  children  and  to  develop  a caring,  prosocial 
context  that  facilitates  educational  processes  in  the  classroom.  Table  6.1  describes  PATHS’  general 
characteristics  (panel  1),  the  types  of  instruction  and  strategies  used  (panel  2),  the  professional  development 
provided  for  those  implementing  the  program  (panel  3),  and  the  social  and  character  development  activities 
(panel  4)  and  outcomes  (panel  5)  addressed  by  the  program.  The  program  includes  the  following 
components: 

• The  PATHS  classroom  curriculum  is  designed  to  facilitate  the  development  of  self-control, 
emotional  understanding,  positive  self-esteem,  relationships,  and  interpersonal  problem-solving  skills 
in  children.  For  20  to  30  minutes  a day,  3 to  5 days  per  week,  teachers  engage  students  in  lessons  that 
involve  discussion,  role-playing,  storytelling,  and  worksheets.  Teachers  are  also  encouraged  to  model 
and  reinforce  social  skills  throughout  the  day. 

• Schoolwide  activities  include  a PATHS  party  at  the  end  of  the  year  and  visual  artifacts  that  illustrate 
aspects  of  social  and  emotional  competence. 

• Parents  are  engaged  in  the  program  through  parent  newsletters  and  homework  assignments  that  are 
to  be  completed  with  their  children. 

• Teachers,  principals,  and  mental  health  staff  participate  in  an  initial  2-day  training  that  presents  the 
concepts  of  program  implementation,  lessons  on  integrating  PATHS  activities  with  traditional 
instruction,  and  methods  for  applying  and  transferring  social  skills  to  children  over  the  entire  school 
day  and  in  the  home.  Program  consultants  provide  weekly  technical  assistance  on  program 
implementation  to  teachers,  a recommended  practice  in  the  PATHS  program.  Consultants  hold 
biweekly  calls  with  teachers  to  discuss  implementation  and  challenges.  In  addition,  a national 
PATHS-certified  trainer  is  available  to  each  intervention  school  1 day  in  the  fall  and  spring.  A 1 -day 
summer  meeting  for  principals,  teachers,  and  school  staff  after  the  first  year  of  implementation 
provides  additional  technical  assistance,  addresses  feedback  on  program  implementation,  and  engages 
school  staff  in  planning  for  the  following  year. 
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Table  6.1 . Promoting  Alternative  Thinking  Strategies 

Panel  1 : General  characteristics 

Target  population 
Universal 

Program  components 
Peer:  In  class 

Parent  Contact  and  involvement 
Classroom:  Lessons 

Schoolwide:  Planned  events  and  program  artifacts 
Community:  None  or  not  major  focus 
Training:  Pretraining  and  ongoing 

Level  of  integration 

Add-on  curriculum  and  schoolwide  activities 
Flexibility 

Manualized:  Scripted  curriculum  and  guidebook 

Adaptability:  Less  adaptable 

See  notes  at  end  of  table. 


Panel  2:  Description  of  instruction  and  strategies 

Classroom 

Lessons 

Who  delivers:  T eacher 

Activities  and  tools:  Direct  instruction,  storytelling,  discussion,  role-playing,  utilization  of  artifacts  (e.g., 
posters, 

turtle  puppet),  worksheets 

Content:  Emotion  understanding  and  control,  behavior  regulation,  problem  solving,  making  friends 
Frequency:  20  to  30  minutes  per  day,  3 to  5 days  per  week 
Strategies 

Who  delivers:  T eacher 

Activities  and  tools:  Modeling  of  skills 

Frequency:  Daily 

Supplement  to  classroom 

Parent  newsletter  and  engagement  in  homework 

Schoolwide  activities 

End-of-year  PATHS  party  and  program  artifacts 

See  notes  at  end  of  table. 
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Table  6.1 . Promoting  Alternative  Thinking  Strategies — Continued 

Panel  3:  Professional  development 

Pre-implementation 

Teachers 

Content  Training  on  concepts,  curriculum  implementation,  how  to  integrate  activities  with  traditional 
instruction 
Duration'.  2 days 
Other 

Content'.  Principal  and  school  mental  health  staff,  same  as  teacher  training 
Duration'.  2 days 

Ongoing  consultation 
Teachers 

Content:  Technical  assistance/consultation  on  implementation;  individual  meetings  with  each  teacher; 

attendance  at  grade-level  meetings;  planning  for  following  year 
Duration:  Weekly  consultations;  2-day  summer  meeting 

Other 

Content:  Principal  and  school  staff;  model  lessons,  team  teaching,  general  feedback;  planning  for 
following  year 

Duration:  Biweekly  calls;  2-day  summer  meeting 

See  notes  at  end  of  table. 


Panel  4:  Activities  for  SACD  goals 


Violence  prevention  and  peace  promotion 

✓ 

Risk  prevention  and  health  promotion 

✓ 

Social  and  emotional  development 

✓ 

Civic  responsibility  and  community  service 

✓ 

Character  education 

✓ 

Behavior  management 

✓ 

Tolerance  and  diversity 

✓ 

See  notes  at  end  of  table. 


Panel  5:  SACD  outcomes  addressed 


Engagement  with  Learning 

✓ 

Empathy 

✓ 

Academic  Competence  and  Motivation 

✓ 

Positive  School  Orientation 

✓ 

Altruistic  Behavior 

Negative  School  Orientation 

✓ 

Positive  Social  Behavior 

✓ 

Student  Afraid  at  School 

Problem  Behavior 

✓ 

Victimization  at  School 

Self-Efficacy  for  Peer  Interactions 

✓ 

Feelings  of  Safety 

✓ 

Normative  Beliefs  About  Aggression 

✓ 

Student  Support  for  Teachers 

NOTE:  Abbreviations  are 
V : Activity  or  outcome  addressed 
Blank  cell:  Activity  or  outcome  not  addressed 
SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Sample  and  Random  Assignment 

The  Children’s  Institute  research  team  recruited  a total  of  10  public  elementary  schools,  representing  1 school 
district  in  Minnesota  (2  schools)  and  2 school  districts  in  New  York  (8  schools).  The  10  schools  were 
randomly  assigned  to  treatment  and  control  conditions  prior  to  the  fall  2004  data  collection.10  A two-step 
process  was  used,  stratified  by  the  three  participating  school  districts.  First,  a computer -generated  pairwise 
matching  algorithm  developed  by  Mathematica  Policy  Research,  Inc.  (MPR)  was  used  to  identify  the  best 
pairwise  matches  across  the  10  schools  based  on  variables  identified  by  the  Children’s  Institute  research  team. 
The  variables  used  in  the  pairwise  matching  for  the  8 New  York  schools  were  as  follows: 

(a)  enrollment;  (b)  percentage  of  White  students;  (c)  percentage  of  limited-English-proficient  students; 
(d)  student-teacher  ratio;  (e)  percentage  of  students  eligible  for  free  lunch;  (f)  percentage  of  students  eligible 
for  reduced-price  lunch;  (g)  percentage  of  students  passing  English  language  arts;  and  (h)  percentage  of 
students  passing  math.  For  the  2 Minnesota  schools,  the  following  variables  were  used:  (a)  enrollment; 

(b)  percentage  of  White  students;  (c)  number  of  suspensions  per  100  students;  (d)  percentage  of  limited- 
English-proficient  students;  (e)  percentage  of  students  eligible  for  free  lunch;  (f)  percentage  of  students 
eligible  for  reduced-price  lunch;  (g)  stability;  (h)  language  test  scores;  and  (i)  math  test  scores.  Second,  using 
the  flip  of  a coin,  1 school  in  each  matched  pair  was  assigned  to  either  the  intervention  or  control  condition. 
Five  schools  received  the  PATHS  program  and  5 schools  acted  as  control  schools  and  continued  to 
implement  the  social  and  character  development  activities  that  constituted  their  standard  practice. 
Assignment  to  treatment  or  control  condition  was  at  the  school  level  and  therefore  limited  the  risk  of 
contamination  between  treatment  and  control  classrooms. 

The  original  student  sample  (the  cohort  of  students  in  the  third  grade  in  the  10  schools  in  fall  2004) 
numbered  786  students  (377  treatment  and  409  control).  Table  6.2  documents  the  change  in  the  sample  over 
the  three  spring  follow-up  data  collection  periods.  Over  time,  new  entrants  to  the  cohort  became  a larger 
percentage  of  the  sample,  eventually  making  up  30  percent  of  the  sample  by  the  spring  of  Year  3.  There  were 
no  statistically  significant  differences  between  the  treatment  and  control  groups  in  the  number  of  new 
entrants.  The  percentage  of  the  sample  made  up  of  the  original  cohort  further  declined  as  students  left  the 
schools.  By  Year  3,  approximately  31  percent  of  the  original  sample  had  left. 


50  In  Year  2 (fall  2005),  the  Children’s  Institute  research  team  recruited  four  more  schools,  two  assigned  to  treatment  and 
two  assigned  to  control,  which  were  followed  to  the  end  of  the  study  (spring  2007).  A description  of  this  second  cohort 
and  all  relevant  findings  can  be  found  in  appendix  A.  The  data  from  this  second  cohort  are  not  included  in  the  analyses 
and  results  reported  in  this  chapter. 
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Year  1 Year  1 Year  2 Year  3 


(Fall  3rd  grade) (Spring  3rd  grade)  (Spring  4th  grade)  (Spring  5th  grade) 

Treat-  Treat-  Treat-  Treat- 


Characteristic 

Total 

ment 

Control 

Total 

ment 

Control 

Total 

ment 

Control 

Total 

ment 

Control 

School  sample  size 

10 

5 

5 

10 

5 

5 

10 

5 

5 

10 

5 

5 

Student  sample  size 

786 

377 

409 

783 

374 

409 

778 

373 

405 

778 

378 

400 

Stayers 

t 

t 

t 

747 

356 

391 

636 

311 

325 

543 

263 

280 

New  entrants 

t 

t 

t 

36 

18 

18 

142 

62 

80 

235 

115 

120 

New  entrants  as  a percent  of 
spring  enrollment 

t 

t 

t 

4.6 

4.8 

4.4 

18.3 

16.6 

19.8 

30.2 

30.4 

30.0 

Total  leavers  (from  original  cohort) 

t 

t 

t 

39 

21 

18 

150 

66 

84 

243 

114 

129 

Leavers  as  percent  of  fall  2004 
enrollment 

t 

t 

t 

5.0 

5.6 

4.4 

19.1 

17.5 

20.5 

30.9 

30.2 

31.5 

Number  of  students  per  school 
(mean) 

79 

75 

82 

78 

75 

82 

78 

75 

81 

78 

76 

80 

Range  of  number  of  students  per 
school 

64-108 

64-83 

65-108 

63-105 

63-82 

64-105 

56-102 

68-85 

56-102 

49-97 

66-90 

49-97 

t Not  applicable. 

NOTE:  No  statistically  significant  differences  were  found  between  values  for  treatment  and  control  groups. 
SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Implementation 

Training 

The  intervention  teachers  received  2 days  of  program  implementation  training  by  PATHS-certified  trainers 
prior  to  the  beginning  of  the  school  year  (table  6.1,  panel  3).  Teachers  had  access  to  ongoing  program 
implementation  support  throughout  the  school  year.  Three  program  consultants  went  to  schools  at  least  once 
a week  to  visit  classrooms,  teach  lessons,  team  teach,  provide  feedback  to  teachers,  answer  questions,  and 
help  with  implementation  as  needed.  These  consultants  were  trained  by  the  program  developers.  Each 
consultant  was  selected  because  of  prior  teaching  experience  and  an  ability  to  form  close  working 
relationships  with  teachers,  students,  and  administrators  at  the  schools.  One  consultant  had  extensive 
classroom  experience,  a second  had  both  classroom  and  administrative  experience,  and  the  third  had 
classroom  experience  as  well  as  experience  as  a volunteer  working  in  an  urban  community.  In  addition,  a 
program  trainer  was  made  available  to  each  intervention  school  during  the  initial  2-day  training  and  during  the 
1-day  summer  meeting  to  answer  questions,  demonstrate  lessons,  provide  feedback,  and  help  with  planning 
for  the  upcoming  year.  In  addition,  the  lead  national  PATHS  trainer  participated  in  the  biweekly  calls 
approximately  50  percent  of  the  time  and  made  occasional  visits  to  the  schools  during  each  school  year. 

Data  Collection 

MPR  collected  the  multiprogram  child,  teacher,  and  school  data  at  the  Minnesota  site,  and  MPR’s 
subcontractor,  Decision  Information  Resources,  Inc.  (DIR),  collected  the  multiprogram  child,  teacher,  and 
school  data  at  the  New  York  site.  Table  6.3  shows  the  school  year  milestones  and  dates  of  implementation 
for  the  New  York/Minnesota  site.  Data  were  collected  in  the  fall  and  spring  of  the  first  2 years  and  the  spring 
of  Year  3.  The  fall  2004  multiprogram  data  collection  began  on  October  18,  2004,  and  ended  on  October  28, 
2004.  The  average  time  frame  from  the  beginning  of  program  implementation  to  the  beginning  of  fall  data 
collection  was  6 weeks.  As  a result,  initial  data  collection  took  place  after  implementation  of  the  PATHS 
program  began.  Therefore,  these  data  provide  a measure  of  the  students,  teachers,  and  schools  near  the 
beginning  of  the  school  year,  at  a time  when  the  PATHS  program  had  been  operating  for  a relatively  short 
period  of  time.  The  spring  data  collection  window  was  from  March  28,  2005,  to  April  28,  2005.  PATHS  had 
been  implemented  for  29  weeks  at  the  time  of  the  spring  data  collection  and  for  22  weeks  from  the  end  of  the 
fall  data  collection.  Year  2 followed  a similar  pattern,  with  implementation  occurring  at  the  start  of  the  school 
year,  fall  data  collection  occurring  5 weeks  later,  and  spring  data  collection  occurring  20  weeks  after  fall  data 
collection  (and  29  weeks  after  the  start  of  implementation).  In  spring  2007,  data  collection  occurred  29  weeks 
after  the  start  of  implementation.  Data  collection  took  between  3 to  5 weeks  at  each  collection  point. 
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Table  6.3.  Data  collection  dates — PATHS 


Year  1 Year  1 Year  2 Year  2 Year  3 

Data  collection  schedule (Fall  3rd  grade)  (Spring  3rd  grade)  (Fall  4th  grade)  (Spring  4th  grade)  (Spring  5th  grade) 


School  sample  size 

10 

10 

10 

10 

10 

School-year  dates 
First  day  of  school 

9/7/04 

t 

9/7/05 

t 

9/6/0  6 1 

Start  of  implementation 

9/8/04 

t 

First  day 

t 

First  day 

Last  day  of  school 

t 

6/23/05 

t 

6/22/06 

6/6/07; 

6/21/072 

Data  collection 
Start 

10/18/04 

3/28/05 

10/11/05 

3/20/06 

3/26/07 

End 

1 0/28/04 

4/28/05 

10/27/05 

4/11/06 

4/27/07 

Calendar  weeks  from 
program  implementation 
to  start  of  fall  2004 
data  collection 

6 

t 

t 

t 

t 

Calendar  weeks  from 
start  of  school  to  start  of 
fall  2004  data  collection 

6 

t 

5 

t 

t 

Calendar  weeks  from 
end  of  fall  data  collection 
to  start  of  spring  data 
collection 

t 

22 

t 

20 

t 

Calendar  weeks  from 
program  implementation 
to  start  of  spring  data 
collection 

t 

29 

t 

29 

291 

t Not  applicable. 

1 Schools  started  between  9/5/06  and  9/7/06.  In  calculations  of  time  from  the  start  of  the  school  year  to  the  start  of  spring  2007  data 
collection  9/6/06  was  used  as  the  first  day  of  the  school  year. 

2 Two  schools  ended  on  6/6/07  and  eight  ended  on  6/21/07. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 


Consent  Rates,  Completion  Rates,  and  Percentage  of  Sample  With  Data 

The  actual  number  of  student,  primary  caregiver,  and  teacher  reports  available  for  analysis  was  smaller  than 
the  number  in  the  sample  because  consent  and  completion  rates  were  below  100  percent.  Primary  caregivers 
had  to  provide  consent  before  children  could  complete  the  Child  Report,  before  their  child’s  teacher  could 
complete  the  Teacher  Report  on  Student,  and  before  they  themselves  completed  the  Primary  Caregiver 
Report.  In  the  spring  of  Years  1 and  3,  significantly'  more  primary  caregivers  in  the  treatment  group  than  in 
the  control  group  provided  consent  for  their  children  and  themselves.  Teachers  also  had  to  provide  consent 
before  completing  the  Teacher  Report  on  Classroom  and  School.  In  Year  1,  significantly'  more  teachers  in  the 
treatment  group  gave  consent  for  both  the  fall  and  spring  data  collection. 

Of  those  with  consent,  not  all  completed  their  respective  reports.  Table  6.4  shows  the  consent  rates, 
completion  rates,  and  percentages  of  sample  with  data  for  each  of  the  four  reports  over  the  3 years.  For  the 
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Child  Report  and  two  teacher  reports,  completion  rates  ranged  from  89  percent  to  100  percent,  with  one 
statistically  significant  difference  in  completion  rates  between  treatment  and  control  groups  for  the  Child 
Report  in  the  spring  of  2005.  In  this  case,  the  treatment  group  had  fewer  completed  reports  than  the  control 
group  (94%  versus  98%).  For  the  Teacher  Report  on  Student,  completion  rates  ranged  from  96  percent  to 
100  percent.  For  the  Primary  Caregiver  Report,  the  completion  rates  dropped  over  time  from  95  percent  to 
79  percent.  There  was  a statistically  significant  difference  for  the  Primary  Caregiver  Report  in  Year  2,  with 
more  caregivers  in  the  treatment  group  completing  reports  compared  to  the  control  group  (83%  versus  79%). 

The  percentages  of  the  sample  with  Child  Report  data  ranged  from  50  percent  to  60  percent  over  the  3 years, 
with  a statistically  significant  difference  in  Year  3,  when  more  students  in  the  treatment  group  had  data  than 
did  students  in  the  control  group  (60%  versus  52%).  The  percentages  of  students  with  information  from  the 
Teacher  Report  on  Student  ranged  from  53  percent  to  62  percent,  with  statistically  significant  differences 
between  treatment  and  control  conditions  in  Years  1 and  3,  both  favoring  the  treatment  group  (62%  versus 
55%  and  60%  versus  53%).  The  percentages  of  students  with  data  from  the  Primary  Caregiver  Report  ranged 
from  42  percent  to  56  percent.  In  general,  this  percentage  declined  over  time,  although  the  highest  rates  were 
seen  in  the  spring  of  2005.  In  Year  2,  the  treatment  group  had  significantly  more  Primary  Caregiver  Report 
data  than  did  the  control  group  (50%  versus  42%).  The  percentages  of  teachers  with  data  from  the  Teacher 
Report  on  Classroom  and  School  ranged  from  70  percent  to  92  percent,  with  a statistically  significant 
difference  in  Year  1,  when  treatment  teachers  had  more  data  than  control  teachers  in  both  the  fall  (89% 
versus  70%)  and  spring  (92%  versus  70%). 


354 


355 


Table  6.4.  Consent  rates,  completion  rates,  and  percentage  of  sample  with  data  from  each  report — PATHS 


Year  1 Year  1 Year  2 Year  3 

(Fall  3rd  grade)  (Spring  3rd  grade)  (Spring  4th  grade)  (Spring  5th  grade) 

Treat-  Treat-  Treat-  Treat- 


Report 

Total 

ment 

Control 

Total 

ment 

Control 

Total 

ment 

Control 

Total 

ment 

Control 

Student  sample  size 

786 

377 

409 

783 

374 

409 

778 

373 

405 

778 

378 

400 

Child  Report  (percent) 

Primary  caregiver  consent  rate 

58.4 

60.5 

56.5 

58.5 

62.3* 

55.0 

57.6 

60.9 

54.6 

56.6 

60.3* 

53.0 

Student  completion  rate 

89.3 

89.9 

88.7 

95.9 

94.0* 

97.8 

96.9 

96.9 

96.8 

98.0 

98.7 

97.2 

Students  with  data1 

52.2 

54.4 

50.1 

56.1 

58.6 

53.8 

55.8 

59.0 

52.8 

55.4 

59.5* 

51.5 

Primary  Caregiver  Report  (percent) 

Primary  caregiver  consent  rate 

56.6 

58.9 

54.5 

57.3 

61.2* 

53.8 

56.0 

59.5 

52.8 

55.1 

58.7* 

51.8 

Primary  caregiver  completion  rate 

94.6 

95.0 

94.2 

84.2 

86.5 

81.8 

81.2 

83.3* 

79.0 

78.8 

77.0 

80.7 

Primary  caregivers  with  data1 

53.6 

56.0 

51.3 

48.3 

52.9 

44.0 

45.5 

49.6*’ 

* 41.7 

43.4 

45.2 

41.8 

Teacher  Report  on  Student  (percent) 

Primary  caregiver  consent  rate2 

58.4 

60.5 

56.5 

58.5 

62.3* 

55.0 

57.6 

60.9 

54.6 

56.6 

60.3* 

53.0 

Teacher  completion  rate 

96.5 

96.9 

96.1 

99.8 

99.6 

100.0 

100.0 

100.0 

100.0 

99.5 

100.0 

99.1 

Students  with  data1 

56.4 

58.6 

54.3 

58.4 

62.0* 

55.0 

57.6 

60.9 

54.6 

56.3 

60.3* 

52.5 

Teacher  Report  on  Classroom  and  School 
(3rd-  to  5th-grade  teachers)  (percent) 

Teacher  consent  rate 

83.8 

93.7** 

75.3 

83.8 

95.2** 

74.0 

84.3 

90.0 

79.7 

93.2 

90.0 

95.9 

Teacher  completion  rate 

93.9 

94.9 

92.7 

95.6 

96.7 

94.4 

99.1 

100.0 

98.3 

89.5 

90.7 

88.6 

Teachers  with  data1 

78.7 

88.9** 

69.9 

80.1 

92.1** 

69.9 

83.6 

90.0 

78.4 

83.5 

81.7 

84.9 

* Treatment  group  significantly  different  from  control  group  at  the  .05  level. 

**  Treatment  group  significantly  different  from  control  group  at  the  .01  level. 

1 Calculated  as  consent  rate  x completion  rate. 

2The  primary  caregiver  consent  rates  for  the  Child  Report  and  the  Teacher  Report  on  Student  are  identical,  as  the  primary  caregiver  gave  consent  to  both  together. 
SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Responses  from  students  in  the  original  cohort  (stayers)  and  new  entrants  in  the  PATHS  sample  were 
examined  to  investigate  possible  differences  between  the  two  groups  in  consent  rates,  completion  rates,  and 
the  percentages  of  the  sample  with  data  that  might  affect  outcome  data  (table  6.5).  In  Years  1 and  3,  consent 
rates  for  stayers  were  significantly  higher  than  for  new  entrants  (by  16  to  26  percentage  points).  These 
differences  in  consent  rates  led  to  similar  differences  between  stayers  and  new  entrants  in  percentages  of  the 
sample  with  data  on  all  three  reports  (by  14  to  26  percentage  points).  In  Year  2,  stayers  had  a significandy 
higher  percentage  of  Primary  Caregiver  report  data  than  new  entrants  (by  10  percentage  points). 
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Table  6.5.  Consent  rates,  completion  rates,  and  percentage  of  sample  with  data:  Stayers  versus  new  entrants — PATHS 

Year  1 Year  2 Year  3 

(Spring  3rd  grade)  (Spring  4th  grade)  (Spring  5th  grade) 

New  New  New 


Report 

Total 

Stayers 

entrants 

Total 

Stayers 

entrants 

Total 

Stayers 

entrants 

Student  sample  size 

783 

747 

36 

778 

636 

142 

778 

543 

235 

Child  Report  (percent) 

Primary  caregiver  consent  rate 

58.5 

59.7** 

33.3 

57.6 

59.1 

50.7 

56.6 

61.7*** 

44.7 

Student  completion  rate 

95.9 

95.7 

100.0 

96.9 

96.3 

100.0 

98.0 

97.9 

98.1 

Students  with  data1 

56.1 

57.2** 

33.3 

55.8 

56.9 

50.7 

55.4 

60.4*** 

43.8 

Primary  Caregiver  Report  (percent) 

Primary  caregiver  consent  rate 

57.3 

58.5** 

33.3 

56.0 

57.5 

49.3 

55.1 

60.0*** 

43.8 

Primary  caregiver  completion  rate 

84.2 

84.4 

75.0 

81.2 

82.2 

75.7 

79.3 

79.8 

77.7 

Primary  caregivers  with  data1 

48.3 

49.4** 

25.0 

45.5 

47.3* 

37.3 

43.7 

47.9** 

34.0 

Teacher  Report  on  Student  (percent) 

Primary  caregiver  consent  rate2 

58.5 

59.7** 

33.3 

57.6 

59.1 

50.7 

56.6 

61.7*** 

44.7 

Teacher  completion  rate 

99.8 

99.8 

100.0 

100.0 

100.0 

100.0 

99.5 

99.7 

99.0 

Students  with  data1 

58.4 

59.6** 

33.3 

57.6 

59.1 

50.7 

56.3 

61.5*** 

44.3 

* Stayers  significantly  different  from  new  entrants  at  the  .05  level. 

**  Stayers  significantly  different  from  new  entrants  at  the  .01  level. 

***  Stayers  significantly  different  from  new  entrants  at  the  .001  level. 

1 Calculated  as  consent  rate  x completion  rate. 

2The  primary  caregiver  consent  rates  for  the  Child  Report  and  the  Teacher  Report  on  Student  are  identical,  as  the  primary  caregiver  gave  consent  to  both  together. 
SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Fidelity  of  Implementation 

Each  year,  PATHS’  five  treatment  schools  were  independently  rated  for  quantity  and  quality  of  program 
implementation  by  two  raters  from  the  research  team.  The  global  measure  of  fidelity  for  the  multisite  study 
was  used;  inter-rater  reliability  was  calculated  using  Cronbach’s  alpha  (Year  1 = 0.78,  Year  2 = 0.93,  and  Year 
3 = 0.86).  The  ratings  were  used  to  generate  a single  consensus  rating  with  which  schools  were  identified  as 
being  high  or  low  in  implementation  fidelity.  In  Year  1,  three  treatment  schools  were  identified  as  having  high 
fidelity,  and  in  Years  2 and  3,  two  treatment  schools  were  identified  as  having  high  fidelity.  Cohen’s  kappa 
was  used  as  the  measure  of  agreement  when  identifying  schools  as  high  fidelity,  and  it  equaled  0.55  in  Year  1, 
0.17  in  Year  2,  and  0.29  in  Year  3. 

Initial  Characteristics 

This  section  examines  the  initial  characteristics  of  the  students,  teachers,  and  schools  participating  in  the 
evaluation  of  the  PATHS  program.  These  characteristics  were  collected  from  students  who  were  enrolled  in 
the  third  grade  at  the  study  schools  in  fall  2004,  as  well  as  from  their  primary  caregivers  and  third-grade 
teachers.  Third-,  fourth-,  and  fifth-grade  teachers  and  principals  in  the  study  schools  also  provided 
information  about  activities  related  to  social  and  character  development  in  these  schools.  Documenting  the 
characteristics  of  students,  teachers,  and  schools  and  initial  measures  of  key  outcomes  at  a point  before  the 
interventions  had  been  operating  for  an  extended  period  helped  to  determine  whether  the  random 
assignment  of  schools  to  treatment  and  control  status  produced  treatment  and  control  groups  with  similar 
distributions  of  observed  characteristics.  As  noted  in  the  following  discussion,  there  were  10  significant 
differences  in  the  observed  characteristics,  8 of  which  (out  of  62  comparisons,  with  3 expected  to  be 
significant  by  chance)  reflected  differences  between  the  treatment  and  control  students,  teachers,  and  schools 
in  the  use  of  SACD  activities  in  the  classroom  and  school. 

Characteristics  of  Children,  Their  Families,  and  Communities 

There  were  no  significant  differences  between  the  treatment  and  control  groups  in  the  observed  student, 
caregiver,  and  community  characteristics  (table  6.6).  For  students,  the  mean  age  was  8.1  years.  The  sample 
contained  roughly  equal  percentages  of  girls  (58%)  and  boys  (43%).  The  sample  was  ethnically  diverse,  with 
White  non-Hispanic  students  making  up  56  percent  of  the  sample,  Black  non-Hispanic  students  making  up 
22  percent  of  the  sample,  and  Hispanic  students  making  up  12  percent  of  the  sample. 

The  sample  was  also  diverse  in  its  levels  of  family  income,  education  levels  of  primary  caregivers  of  the 
children  in  the  sample,  and  family  situation.  For  the  total  sample,  34  percent  of  children  lived  in  a household 
where  the  income  was  135  percent  of  the  federal  poverty  level  or  lower,  which  is  the  income  threshold  for 
eligibility  for  free  school  meals.  Thirteen  percent  of  primary  caregivers  had  not  completed  high  school.  Nearly 
two-thirds  of  the  children  (66%)  lived  with  both  their  mother  and  their  father.  There  were  no  significant 
differences  between  the  treatment  and  control  groups  in  these  characteristics. 

The  mean  values  of  the  outcome  measures  for  children’s  behavior  and  attitudes  as  reported  by  the  primary 
caregiver,  child,  and  teacher  at  initial  data  collection  in  fall  2004  are  shown  in  table  6.7.  There  was  1 
significant  difference  between  the  treatment  and  control  groups  (out  of  18  comparisons):  children  in  the 
treatment  group  reported  lower  Self-Efficacy  for  Peer  Interaction  (2.8  versus  3.0). 
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Table  6.6.  Initial  characteristics  of  children,  their  families,  and  communities — PATHS 


Characteristic 

Total 

Treatment 

Control 

Student  sample  size 

421 

211 

210 

Student  demographics 

Gender  (percent) 

Male 

42.5 

40.5 

44.6 

Female 

57.5 

59.5 

55.4 

Race/ethnicity  (percent) 

White  (non-Hispanic) 

56.1 

52.7 

59.4 

Black  (non-Hispanic) 

22.3 

23.9 

20.7 

Hispanic 

11.5 

12.3 

10.7 

Other 

10.1 

11.0 

9.2 

Age  (in  years)  (mean) 

8.1 

8.1 

8.1 

Primary  caregiver  and  family  characteristics 

Primary  caregiver’s  age  (in  years)  (mean) 

37.1 

36.6 

37.6 

Primary  caregiver’s  race/ethnicity  (percent) 

White  (non-Hispanic) 

63.2 

60.9 

65.5 

Black  (non-Hispanic) 

20.6 

22.9 

18.4 

Hispanic 

9.9 

9.7 

10.2 

Other 

6.3 

6.6 

6.0 

Primary  caregiver’s  education  (percent) 

Did  not  complete  high  school 

12.7 

13.0 

12.4 

Completed  high  school  or  equivalent 

18.5 

19.0 

17.9 

Some  college 

37.1 

39.4 

34.8 

Bachelor’s  or  higher  degree 

31.7 

28.5 

34.8 

Primary  caregiver’s  employment  (percent) 

Full-time 

51.4 

56.7 

46.0 

Other 

48.6 

43.3 

54.0 

Primary  caregiver’s  marital  status  (percent) 

Married 

62.9 

63.3 

62.5 

Other 

37.1 

36.7 

37.5 

Students  who  live  in  one  household  (percent) 

93.7 

92.8 

94.5 

Number  of  individuals  in  household  (mean) 

4.6 

4.6 

4.6 

Primary  caregiver’s  relationship  to  child  (percent) 

Mother  (stepmother) 

86.1 

86.0 

86.2 

Father  (stepfather) 

10.3 

10.5 

10.1 

Other  relative/nonrelative 

3.6 

3.5 

3.7 

See  notes  at  end  of  table. 
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Table  6.6.  Initial  characteristics  of  children,  their  families,  and  communities — PATHS — 
Continued 


Characteristic 

Total 

Treatment 

Control 

Student  lives  with  (percent) 

Mother  (stepmother)  and  father  (stepfather) 

66.3 

65.7 

66.9 

Mother  (stepmother)  only;  father  (stepfather)  not  present 

29.2 

28.9 

29.4 

Father  (stepfather)  only;  mother  (stepmother)  not  present 

1.6 

t 

t 

Other  relative/nonrelative,  parents  not  present 

2.9 

3.1 

2.7 

Highest  education  of  anyone  in  household  (percent) 

Did  not  complete  high  school 

8.2 

8.9 

7.4 

Completed  high  school  or  equivalent 

18.4 

17.4 

19.4 

Some  college 

33.8 

39.4 

28.2 

Bachelor’s  or  higher  degree 

39.7 

34.3 

45.0 

Total  household  income  (percent) 

Less  than  $20,000 

29.1 

25.9 

32.2 

$20,000  to  $39,999 

18.3 

22.1 

14.4 

$40,000  to  $59,999 

17.2 

23.3 

11.0 

$60,000  or  more 

35.5 

28.7 

42.3 

Income-to-poverty-threshold  ratio — Below  135  percent  (percent) 

33.6 

31.7 

35.5 

Income-to-poverty-threshold  ratio — 135  to  185  percent  (percent) 

14.1 

16.6 

11.6 

Income-to-poverty-threshold  ratio — Above  185  percent  (percent) 

52.3 

51.7 

52.8 

Alabama  Parenting  Questionnaire — Poor  Monitoring  and 

Supervision  Subscale  (mean) 

1.1 

1.1 

1.1 

Alabama  Parenting  Questionnaire — Positive  Parenting 

Subscale  (mean) 

3.5 

3.5 

3.5 

Confusion,  Hubbub,  and  Order  Scale  (mean) 

2.2 

2.2 

2.2 

Community  characteristics  (mean) 

Community  Risks  Scale 

1.4 

1.3 

1.5 

Community  Resources  Scale 

3.0 

2.9 

3.0 

Child-Centered  Social  Control  Scale 

3.1 

3.0 

3.2 

$ Reporting  standards  not  met.  Values  suppressed  to  protect  confidentiality. 

NOTE:  No  statistically  significant  differences  were  found  between  values  for  treatment  and  control  groups.  Weights,  which  assign 
equal  weight  to  each  school  within  the  program,  were  used  in  producing  the  treatment,  control,  and  overall  means.  Statistical  tests 
were  conducted  using  regressions  that  included  program  indicators  to  account  for  the  sample  design  and  adjusted  for  clustering  at 
the  school  level. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Table  6.7.  Mean  scores  and  standard  deviations  for  initial  outcome  measures  of  sample — 
PATHS 


Total  Treatment  Control 


Outcome  measure-Report 

Range 

Mean 

SD 

Mean 

SD 

Mean 

SD 

Social  and  Emotional  Competence  Domain 

Self-Efficacy  for  Peer  Interaction-CR 

1-4 

2.9 

0.6 

2.8* 

0.6 

3.0 

0.6 

Normative  Beliefs  About  Aggression-CR 

1-4 

1.2 

0.3 

1.2 

0.4 

1.1 

0.3 

Empathy-CR 

1-4 

2.4 

0.4 

2.4 

0.4 

2.5 

0.4 

Behavior  Domain 

Altruistic  Behavior-CR 

0-3 

1.3 

0.9 

1.4 

0.9 

1.3 

0.8 

Altruistic  Behavior-TRS 

1-4 

1.2 

0.3 

1.2 

0.3 

1.3 

0.3 

Altruistic  Behavior-PCR 

1-4 

2.2 

0.7 

2.3 

0.7 

2.2 

0.8 

Positive  Social  Behavior-TRS 

1-4 

3.0 

0.7 

3.0 

0.7 

3.1 

0.6 

Positive  Social  Behavior-PCR 

1-4 

3.0 

0.5 

3.0 

0.5 

3.0 

0.5 

Problem  Behavior-CR 

0-3 

0.2 

0.3 

0.2 

0.4 

0.2 

0.3 

Problem  Behavior-TRS 

1-4 

1.3 

0.5 

1.4 

0.5 

1.3 

0.4 

Problem  Behavior-PCR 

1-4 

1.6 

0.3 

1.6 

0.3 

1.6 

0.3 

ADHD-Related  Behavior-TRS 

1-4 

1.8 

0.7 

1.8 

0.7 

1.7 

0.7 

Academics  Domain 

Academic  Competence  and  Motivation-TRS 

1-5 

3.0 

0.9 

2.9 

0.9 

3.2 

0.9 

Engagement  with  Learning-CR 

1-4 

3.8 

0.5 

3.7 

0.5 

3.8 

0.4 

Perceptions  of  School  Climate  Domain 

Positive  School  Orientation-CR 

1-4 

3.3 

0.6 

3.3 

0.6 

3.4 

0.5 

Negative  School  Orientation-CR 

1-4 

1.7 

0.6 

1.8 

0.6 

1.7 

0.6 

Student  Afraid  at  School-CR 

1-4 

2.4 

0.9 

2.5 

0.9 

2.3 

0.9 

Victimization  at  School-CR 

0-3 

0.7 

0.8 

0.7 

0.8 

0.7 

0.8 

Student  sample  size — PCR 

421 

211 

210 

Student  sample  size — CR 

410 

205 

205 

Student  sample  size — TRS 

443 

221 

222 

* Treatment  group  significantly  different  from  control  group  at  the  .05  level. 

NOTE:  Abbreviations  are 
CR:  Child  Report 
PCR:  Primary  Caregiver  Report 
TRS:  Teacher  Report  on  Student 
ADHD:  Attention  deficit  hyperactivity  disorder 
SD:  Standard  deviation 

Weights,  which  assign  equal  weight  to  each  school  within  the  program,  were  used  in  producing  the  treatment,  control,  and  overall 
means.  Statistical  tests  were  conducted  using  regressions  that  included  program  indicators  to  account  for  the  sample  design  and 
adjusted  for  clustering  at  the  school  level.  Sample  size  may  differ  for  some  outcomes  due  to  nonresponse. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 


Characteristics  of  Teachers  and  Schools 

Table  6.8  describes  the  third-,  fourth-,  and  fifth-grade  teachers  at  the  study  schools.  The  majority  were  White 
non-Hispanic  (88%).  Most  of  the  teachers  were  female  (74%),  and  there  was  a statistically  significant 
difference  in  the  percentages  of  male  teachers,  with  fewer  males  in  the  treatment  group  than  in  the  control 
group  (17%  versus  35%).  Overall,  the  teachers  had  an  average  of  11.5  years  of  total  teaching  experience.  The 
majority  (77%)  held  an  advanced  or  specialist  degree. 
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Data  regarding  school  characteristics  were  drawn  from  the  Common  Core  of  Data  in  order  to  compare 
treatment  and  control  schools.  There  were  no  significant  differences  between  the  two  groups  of  schools  in 
terms  of  student  composition  (race/ ethnicity  and  school  lunch  eligibility),  number  of  students  enrolled, 
number  of  full-time  teachers,  Title  I status,  or  number  of  years  the  principal  had  been  at  the  school  (see  table 
6.9).  In  addition,  there  were  no  significant  differences  between  treatment  and  control  schools  in  terms  of 
location  (urban,  suburban,  or  rural)  or  lowest  and  highest  grade  offered  (these  data  are  not  shown  in  a table). 


Table  6.8.  Initial  characteristics  of  teachers  in  sample — PATHS 


Characteristic 

Total 

Treatment 

Control 

Teacher  sample  size 

107 

56 

51 

Gender  (percent) 

Male 

26.1 

17.4* 

34.8 

Female 

73.9 

82.6 

65.2 

Race/ethnicity  (percent) 

White  (non-Hispanic) 

87.7 

88.7 

86.8 

Other 

12.3 

11.3 

13.2 

Number  of  years  teaching  experience  (mean) 

11.5 

10.0 

13.0 

Number  of  years  teaching  experience  in  this  school  (mean) 

7.7 

6.5 

8.9 

Type  of  teaching  certificate  (percent) 

Regular  state  certificate  or  advanced  professional  certificate 

81.0 

82.4 

79.6 

Other 

19.0 

17.6 

20.4 

Education  (percent) 

Bachelor’s  degree 

23.0 

19.9 

26.2 

Advanced  degree/other 

77.0 

80.1 

73.8 

* Treatment  group  significantly  different  from  control  group  at  the  .05  level. 

NOTE:  Weights,  which  assign  equal  weight  to  each  school  within  the  program,  were  used  in  producing  the  treatment,  control,  and 
overall  means.  Statistical  tests  were  conducted  using  regressions  that  included  program  indicators  to  account  for  the  sample  design 
and  adjusted  for  clustering  at  the  school  level.  Sample  size  may  differ  for  some  outcomes  due  to  nonresponse. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Table  6.9.  Initial  characteristics  of  schools  in  sample — PATHS 


Characteristic 

Total 

Treatment 

Control 

School  sample  size 

10 

5 

5 

Student  race/ethnicity  (percent) 

White  (non-Hispanic) 

46.0 

48.5 

43.6 

Black  (non-Hispanic) 

35.6 

31.1 

40.2 

Hispanic 

13.1 

14.4 

11.8 

Other 

5.2 

5.9 

4.4 

Students  eligible  for  free  or  reduced-price  lunch  (percent) 

56.4 

53.7 

59.0 

Number  of  students  enrolled  (mean) 

538.0 

460.6 

634.8 

Number  of  full-time  teachers  (mean) 

45.1 

42.8 

47.4 

Title  1 status  (percent) 

Title  1 eligible  school 

60.0 

60.0 

60.0 

Schoolwide  Title  1 

66.7 

66.7 

66.7 

Number  of  years  principal  has  been  at  this  school  (mean) 

5.8 

4.0 

7.6 

NOTE:  No  statistically  significant  differences  were  found  between  values  for  treatment  and  control  groups.  Weights,  which  assign 
equal  weight  to  each  school  within  the  program,  were  used  in  producing  the  treatment,  control,  and  overall  means.  Statistical  tests 
were  conducted  using  regressions  that  included  program  indicators  to  account  for  the  sample  design  and  adjusted  for  clustering  at 
the  school  level.  Sample  size  may  differ  for  some  outcomes  due  to  nonresponse. 

SOURCE:  NCES  Common  Core  of  Data  (2003-04),  the  Social  and  Character  Development  (SACD)  Research  Program. 


In  the  Teacher  Report  on  Classroom  and  School,  teachers  reported  on  nine  dimensions  of  school 
environment  (these  data  are  not  shown  in  a table):  feelings  of  safety,  adequacy  of  resources,  student  support, 
freedom  to  teach  as  desired,  affiliation  with  and  ties  to  colleagues,  innovation  regarding  new  approaches  to 
teaching,  professional  interest,  participatory  decisionmaking,  and  work  pressure.  There  were  no  statistically 
significant  differences  between  treatment  and  control  schools  in  these  reports. 

The  Level  of  SACD  in  the  Schools  Near  the  Beginning  of  the  Study 

During  the  initial  data  collection  period,  principals  and  teachers  reported  on  the  SACD  activities  used  in  the 
schools  and  classrooms,  the  availability  of  SACD  materials,  and  the  professional  development  provided  on 
SACD.  Table  6.10  shows  that  the  majority  of  the  school  principals  reported  activities  to  promote  six  social 
and  character  development  goals:  violence  prevention  and  peace  promotion  (90%),  social  and  emotional 
development  (89%),  character  education  (90%),  tolerance  and  diversity  (78%),  risk  prevention  and  health 
promotion  (60%),  and  civic  responsibility  and  community  service  (80%).  In  addition,  90  percent  of  the 
principals  reported  activities  directed  toward  behavior  management.  There  were  no  statistically  significant 
differences  between  the  treatment  group  and  the  control  group  in  the  percentages  reported  by  principals, 
although  this  may  be  due  to  the  small  principal  sample  size.  Teachers’  reports  of  the  use  of  these  activities  in 
their  classrooms  ranged  from  53  percent  to  83  percent,  and  there  were  no  significant  differences  between 
treatment  and  control  teachers  in  their  use.  With  respect  to  the  use  of  schoolwide  activities,  65  percent  to  89 
percent  of  teachers  reported  that  their  schools  used  such  activities.  There  were  no  significant  differences 
between  treatment  and  control  teachers  in  reports  of  their  use  of  these  specific  SACD  activities. 
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Table  6.10.  Principal  and  teacher  initial  reports  on  use  of  SACD  programs  or  activities  in 
sample — PATHS 


SACD  program  or  activity 

Total 

Treatment 

Control 

Principal  sample  size 

10 

5 

5 

Teacher  sample  size 

107 

56 

51 

Principals  reporting  that  school  had  programs  or  activities 
to  promote  the  following  SACD  goals  (percent) 

Violence  prevention  and  peace  promotion 

90.0 

100.0 

80.0 

Social  and  emotional  development 

88.9 

100.0 

75.0 

Character  education 

90.0 

100.0 

80.0 

Tolerance  and  diversity 

77.8 

80.0 

75.0 

Risk  prevention  and  health  promotion 

60.0 

60.0 

60.0 

Civic  responsibility  and  community  service 

80.0 

100.0 

60.0 

Behavior  management 

90.0 

100.0 

80.0 

None  of  the  above 

t 

t 

t 

Teachers  reporting  on  using  programs  or  activities  in  their 
class  to  promote  the  following  SACD  goals  (percent) 

Violence  prevention  and  peace  promotion 

74.1 

78.9 

69.2 

Social  and  emotional  development 

79.8 

83.1 

76.5 

Character  education 

83.1 

81.2 

85.0 

Tolerance  and  diversity 

62.7 

59.8 

65.7 

Risk  prevention  and  health  promotion 

52.9 

45.6 

60.3 

Civic  responsibility  and  community  service 

70.1 

65.9 

74.3 

Behavior  management 

80.4 

85.6 

75.2 

None  of  the  above 

3.1 

t 

t 

Teachers  reporting  schoolwide  use  of  the  following 
activities  to  promote  SACD  (percent) 

Morning  announcements  or  videos 

83.7 

77.3 

90.0 

School  assemblies 

89.4 

90.2 

88.6 

School  newspapers  or  bulletins 

72.1 

62.0 

82.2 

Special  school  days 

65.2 

62.9 

67.5 

Special  events 

81.0 

84.5 

77.4 

Other  activities 

16.6 

13.9 

19.2 

4:  Reporting  standards  not  met.  Values  suppressed  to  protect  confidentiality. 

NOTE:  No  statistically  significant  differences  were  found  between  values  for  treatment  and  control  groups.  Weights,  which  assign 
equal  weight  to  each  school  within  each  of  the  programs  and  to  each  program  across  programs,  were  used  in  producing  the 
treatment,  control,  and  overall  means.  Statistical  tests  were  conducted  using  regressions  that  included  program  indicators  to 
account  for  the  sample  design  and  adjusted  for  clustering  at  the  school  level.  Sample  size  may  differ  for  some  outcomes  due  to 
nonresponse. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Teachers  reported  using  a broad  range  of  teaching  materials  to  support  SACD  activities  (table  6.1 1),  including 
teacher  guides  (55%),  student  materials  (51%),  instructional  aids  (28%),  giveaways  (43%),  and  children’s 
literature  (65%).  There  were  statistically  significant  differences  in  the  use  of  two  types  of  SACD  materials  (out 
of  seven  comparisons  made);  more  treatment  than  control  teachers  used  teacher  guides  (67%  versus  43%) 
and  fewer  treatment  than  control  teachers  used  giveaways  (33%  versus  54%). 

Teachers  also  reported  using  a wide  variety  of  teaching  strategies  (table  6.11).  All  teachers  reported  using  any 
of  the  20  strategies  asked  about,  and  teachers  used  an  average  of  1 1.7  of  the  strategies.  There  was  a significant 
difference  in  the  use  of  role-playing  (1  of  20  strategies),  with  treatment  teachers  using  this  strategy  more  often 
than  control  teachers  (81%  versus  44%). 
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Table  6.11.  Teacher  initial  reports  on  use  of  SACD  materials  and  classroom  strategies  in 
sample — PATHS 


SACD  material  and  classroom  strategy 

Total 

Treatment 

Control 

Teacher  sample  size 

107 

56 

51 

Teachers  using  the  following  materials  in  conjunction  with 
social  and  character  development  activities  (percent) 

Teacher  guides  (manuals,  curricula) 

55.0 

67.3* 

42.8 

Student  materials  (workbooks,  worksheets) 

51.1 

55.6 

46.7 

Instructional  aids  (games,  software,  videos) 

28.0 

30.5 

25.6 

Giveaways  (bookmarks,  stickers) 

43.4 

33.0** 

53.9 

Children’s  literature 

65.3 

66.7 

63.9 

Other  types  of  materials 

15.8 

6.5 

25.0 

Do  not  use  any  of  the  materials  listed  above 

10.7 

9.8 

11.7 

Teachers  using  any  of  the  strategies  listed  below  to  promote 
social  and  character  development  in  the  classroom  (percent) 

100.0 

100.0 

100.0 

Number  of  strategies  (listed  below)  used  by  teachers  to  promote 
social  and  character  development  in  the  classroom  (mean) 

11.7 

12.1 

11.2 

Teachers  using  each  of  the  following  strategies  to  promote 
social  and  character  development  (percent) 

Role-playing 

62.5 

80.8** 

44.2 

Cooperative  learning 

94.3 

95.7 

93.0 

Peer  group  discussions 

90.0 

93.0 

87.0 

Direct  instruction  of  social  and  character  development 

82.3 

91.5 

73.1 

Skill  training 

49.6 

52.6 

46.6 

Incorporating  social  and  character  development  into 
academic  curriculum 

67.6 

75.2 

59.9 

Parent  training 

5.1 

t 

t 

Parent/community  involvement  in  program  development  or  delivery 

31.3 

33.8 

28.8 

Mentoring 

25.3 

24.2 

26.4 

Good  behavior  notes  sent  home  daily  or  weekly 

78.5 

79.7 

77.4 

Presenting  role  models 

78.5 

76.0 

80.9 

Targeted  story  reading  or  writing  on  SACD  themes 

73.3 

78.2 

68.4 

Peer  mediation 

35.7 

36.3 

35.0 

Honor  roll  for  positive  behavior 

37.3 

39.3 

35.4 

Pledges  or  recitations  on  social  and  character  development  themes 

40.7 

29.7 

51.7 

Guided  visualization 

44.0 

43.8 

44.2 

Student-led/student-assisted  instruction 

51.6 

59.4 

43.8 

Journaling 

68.6 

69.0 

68.2 

Time  out  for  negative  behavior 

76.1 

81.2 

70.9 

Daily  or  weekly  rewards  for  positive  behavior 

93.1 

92.0 

94.1 

t Reporting  standards  not  met.  Values  suppressed  to  protect  confidentiality. 

* Treatment  group  significantly  different  from  control  group  at  the  .05  level. 

**  Treatment  group  significantly  different  from  control  group  at  the  .01  level. 

NOTE:  Weights,  which  assign  equal  weight  to  each  school  within  each  of  the  programs  and  to  each  program  across  programs,  were 
used  in  producing  the  treatment,  control,  and  overall  means.  Statistical  tests  were  conducted  using  regressions  that  included 
program  indicators  to  account  for  the  sample  design  and  adjusted  for  clustering  at  the  school  level.  Sample  size  may  differ  for  some 
outcomes  due  to  nonresponse. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Principals  and  teachers  reported  on  participation  in  and  amount  of  SACD  training  and  staff  development 
provided  over  the  previous  12  months  (table  6.12).  Principals  reported  higher  participation  rates  (89%  versus 
80%)  and  more  hours  of  training  (10.3  versus  9.5)  than  did  teachers.  There  were  significant  differences 
between  treatment  and  control  principals  on  the  number  of  hours  of  SACD  training  reported  (1 5 versus  5) 
and  between  treatment  and  control  teachers  on  both  the  percentages  reporting  participation  in  SACD 
training  (92%  versus  68%)  and  the  number  of  hours  of  SACD  training  received  (13  versus  6).  Treatment 
teachers  also  reported  more  training  than  control  teachers  on  two  of  the  seven  specific  SACD  goals:  social 
and  emotional  development  (64.8%  versus  15.3%)  and  character  education  (53.9%  versus  17.2%). 


Table  6.12.  Principal  and  teacher  initial  reports  on  use  of  SACD  materials  and  classroom 
strategies  in  sample — PATHS 


SACD  professional  development 

Total 

Treatment 

Control 

Principal  sample  size 

10 

5 

5 

Teacher  sample  size 

107 

56 

51 

Principals  reporting  that  staff  participated  in  social  and  character 
development  training  within  the  past  year  (percent) 

88.9 

100.0 

75.0 

Teachers  reporting  participation  in  social  and  character 
development  training  within  the  past  12  months  (percent) 

79.6 

91.7* 

67.5 

Number  of  hours  of  social  and  character  development  training 
principals  report  were  provided  to  each  staff  person  last  year  (mean) 

10.3 

14.8* 

4.8 

Number  of  hours  of  social  and  character  development  training 
teachers  report  receiving  during  the  past  12  months  (mean) 

9.5 

12.7** 

6.2 

Teachers  reporting  receiving  training  in  the  past  12  months 
in  the  following  areas  (percent) 

Violence  prevention  and  peace  promotion 

36.7 

30.1 

43.4 

Social  and  emotional  development 

40.0 

64.8** 

15.3 

Character  education 

35.5 

53.9* 

17.2 

Tolerance  and  diversity 

18.5 

22.0 

14.9 

Risk  prevention  and  health  promotion 

11.2 

13.6 

8.9 

Civic  responsibility  and  community  service 

8.7 

4.8 

12.6 

Behavior  management 

31.4 

36.2 

26.5 

* Treatment  group  significantly  different  from  control  group  at  the  .05  level. 


**  Treatment  group  significantly  different  from  control  group  at  the  .01  level. 

NOTE:  Weights,  which  assign  equal  weight  to  each  school  within  each  of  the  programs  and  to  each  program  across  programs,  were 
used  in  producing  the  treatment,  control,  and  overall  means.  Statistical  tests  were  conducted  using  regressions  that  included 
program  indicators  to  account  for  the  sample  design  and  adjusted  for  clustering  at  the  school  level.  Sample  size  may  differ  for  some 
outcomes  due  to  nonresponse. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 


The  data  on  the  initial  level  of  SACD  activity  emphasize  that  the  control  condition  was  a “standard  practice” 
control.  Standard  practice  at  the  control  schools  included  using  SACD  activities,  materials,  and  practices, 
along  with  professional  development,  at  rates  and  in  types  and  amounts  similar  to  the  treatment  schools.  For 
example,  the  percentages  of  teachers  who  reported  using  programs  or  activities  to  promote  specific  SACD 
goals  ranged  from  46  percent  to  86  percent  in  the  treatment  schools  and  from  60  percent  to  85  percent  in  the 
control  schools.  However,  the  8 significant  differences  between  the  treatment  and  control  conditions  in  the 
use  of  SACD  activities  was  more  than  expected  by  chance  (3  out  of  62  comparisons),  and  in  7 of  these  cases 


367 


Chapter  6.  Promoting  Alternative  Thinking  Strategies  (PATHS) 


the  differences  favored  the  treatment  group.  This  may  reflect  the  fact  that  program  implementation  and 
program  training  for  staff  started  before  initial  data  collection. 

Impacts  on  Use  of  SACD  Activities 

The  introduction  of  the  formal  PATHS  program  would  be  expected  to  increase  the  use  of  SACD  activities  in 
the  treatment  schools  in  comparison  to  the  control  schools.  The  analysis  of  this  impact  is  based  on  the 
Teacher  Report  on  Classroom  and  School  (TRCS).  Every  spring,  third-,  fourth-,  and  fifth-grade  teachers 
provided  information  through  the  TRCS  about  the  social  and  character  development  activities  they  used  in 
their  classrooms.  Specifically,  information  from  the  TRCS  was  used  to  determine  the  difference  between 
treatment  and  control  teachers  in  these  areas: 

1.  the  use  of  SACD  activities  in  their  classrooms  overall  and  by  SACD  goal; 

2.  the  use  of  materials  and  strategies  to  implement  the  SACD  activities  within  classrooms  and  within 
the  entire  school; 

3.  the  use  of  staff  development  to  support  the  teachers;  and 

4.  staff  attitudes  toward  the  use  of  SACD  activities. 

TRCS  consent  and  completion  rates  (table  6.4)  led  to  70  percent  to  92  percent  of  all  teachers  having  data  for 
the  3 years,  with  a significantly  greater  percentage  of  treatment  than  control  teachers  providing  data  in  Year  1. 
To  estimate  intervention  impacts  for  each  of  the  outcome  measures,  testing  of  the  statistical  significance  of 
the  differences  in  means  was  used.  Preliminary  analysis  indicated  little  or  no  gains  in  precision  from  using 
covariates.  Before  testing  the  mean  differences,  the  data  were  weighted  such  that  each  school  received  equal 
weight.  Standard  errors  of  the  impact  estimates  account  for  the  clustering  of  teachers  within  schools.  In 
addition,  to  estimate  the  impact  on  the  individual  outcome  measures,  a set  of  heuristics  was  applied  to 
determine  whether  each  outcome  domain  was  statistically  significant  after  adjustments  were  made  for  the 
multiple  tests  conducted. 

Use  of  Activities 

The  percentages  of  control  teachers  who  reported  using  any  SACD  activities  in  their  classrooms  ranged  from 
90  percent  to  99  percent  over  the  3 years  (table  6.13,  panel  1).  For  the  six  individual  SACD  goals,  the  ranges 
varied  from  71  percent  to  86  percent  in  Year  1,51  percent  to  74  percent  in  Year  2,  and  59  percent  to  84 
percent  in  Year  3.  Control  teachers’  use  of  behavior  management  activities  ranged  from  89  percent  to  93 
percent  over  this  period.  The  percentages  of  control  teachers  who  reported  using  any  SACD  activities  in  their 
classroom  for  at  least  1 hour  per  week  (panel  2)  ranged  from  46  percent  to  83  percent  over  the  3 years.  For 
the  six  individual  SACD  goals,  the  ranges  varied  from  10  percent  to  25  percent  in  Year  1,  11  percent  to  40 
percent  in  Year  2,  and  6 percent  to  37  percent  in  Year  3.  Control  teachers’  use  of  behavior  management 
activities  ranged  from  71  percent  to  73  percent  over  this  period. 

For  teachers’  reported  use  of  any  SACD  activity  (panels  1 and  2),  48  comparisons  were  made,  with  2 expected 
to  be  significant  by  chance.  There  were  14  significant  impacts  on  teachers’  reported  use  of  specific  SACD 
activities  in  all  3 years.  In  Year  1,  PATHS  had  significant  impacts  on  engagement  in  activities  for  at  least  1 
hour  per  week:  violence  prevention  and  peace  promotion  (impact  = 32  percentage  points),  social  and 
emotional  development  (impact  = 48  percentage  points),  character  education  (impact  = 28  percentage 
points),  and  tolerance  and  diversity  (impact  = 20  percentage  points).  In  Year  2,  there  were  significant  impacts 
both  overall  and  for  1 hour  per  week  on  social  and  emotional  development  (impact  = 38  and  36  percentage 
points),  character  education  (impact  = 23  and  38  percentage  points),  and  tolerance  and  diversity  (impact  = 33 
and  27  percentage  points).  There  was  also  a significant  impact  on  violence  prevention  and  peace  promotion 
overall  (impact  = 23  percentage  points).  In  Year  3,  PATHS  had  an  impact  on  social  and  emotional 
development  overall  (impact  = 24  percentage  points)  and  for  at  least  1 hour  per  week  (impact  = 27 
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percentage  points).  Civic  responsibility  and  community  service  activities  for  at  least  1 hour  per  week  were  also 
significantly  impacted  (impact  =18  percentage  points).  After  the  heuristics  were  applied,  the  domain  for 
engagement  in  SACD  activities  showed  that  PATHS  had  statistically  significant  impacts  in  Years  1 and  2. 

For  teachers’  reported  use  of  any  named  SACD  activity  (panels  3 and  4),  42  comparisons  were  made,  with  2 
expected  to  be  significant  by  chance.  Six  of  the  12  impact  estimates  in  Year  1,  9 of  the  12  in  Year  2,  and  9 of 
the  12  in  Year  3 were  statistically  significant.  In  all  3 years,  the  use  of  named  activities  to  promote  violence 
prevention  and  peace  promotion  (impact  = 41,  50,  and  36  percentage  points),  social  and  emotional 
development  (impact  = 59,  64,  and  53  percentage  points),  character  education  (impact  = 49,  65,  and  40 
percentage  points),  tolerance  and  diversity  (impact  = 42,  31,  and  34  percentage  points),  and  any  named 
activity  (impact  = 44,  37,  and  33  percentage  points)  were  significantly  impacted.  In  Year  2,  named  activities  to 
promote  civic  responsibility'  and  community  service  were  significantly  impacted  (impact  = 13  percentage 
points).  The  use  of  named  activities  for  at  least  1 hour  per  week  showed  significant  impacts  on  the  use  of  any 
named  activity  in  Year  1 (impact  = 42  percentage  points).  In  Years  2 and  3,  PATHS  had  significant  impact  on 
activities  related  to  social  and  emotional  development  (impact  = 43  and  44  percentage  points),  character 
education  (impact  = 43  and  41  percentage  points),  and  tolerance  and  diversity  (impact  = 19  and  24 
percentage  points).  In  Years  1 and  3,  there  were  significant  impacts  on  the  use  of  any  named  activity  for  at 
least  1 hour  per  week  (impact  = 42  and  40  percentage  points).  PATHS  had  a significant  impact  on  the 
domain  of  engagement  in  named  SACD  activities  in  all  3 years. 
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Table  6.13.  Impacts  on  teacher-reported  SACD  classroom  activities — PATHS 

Panel  1 : Engagement  in  any  activities  to  promote  SACD  goals1 

Year  1 Year  2 Year  3 

(Spring  3rd  grade) (Spring  4th  grade) (Spring  5th  grade) 

T reat-  T reat-  T reat- 


SACD  activity 

ment 

Control 

Impact 

p-value 

ment 

Control 

Impact 

p-value 

ment 

Control 

Impact 

p-value 

Teacher  sample  size 

60 

56 

55 

57 

50 

61 

Violence  prevention  and  peace  promotion 
(percent) 

85.0 

78.9 

6.1 

0.554 

85.9* 

62.4 

23.4 

0.043 

88.2 

81.2 

7.1 

0.446 

Social  and  emotional  development 
(percent) 

89.0 

71.5 

17.4 

0.122 

93.5* 

55.1 

38.4 

0.006 

98.0* 

74.5 

23.5 

0.020 

Character  education  (percent) 

87.7 

85.8 

1.9 

0.861 

95.5* 

72.2 

23.3 

0.015 

95. 9A 

83.6 

12.3 

0.091 

Tolerance  and  diversity  (percent) 

78.7 

70.8 

7.9 

0.420 

88.9* 

56.2 

32.7 

0.025 

86.5 

72.3 

14.2 

0.142 

Risk  prevention  and  health  promotion 
(percent) 

68.5 

70.5 

-2.0 

0.916 

74.3 

73.6 

0.8 

0.967 

80.5 

80.4 

0.1 

0.995 

Civic  responsibility  and  community  service 
(percent) 

65.4 

71.7 

-6.3 

0.566 

67.3 

51.2 

16.1 

0.199 

66.4 

59.4 

7.0 

0.623 

Any  SACD  goal  (percent) 

94.3 

93.7 

0.5 

0.922 

97.0 

90.1 

6.8 

0.185 

100.0 

98.5 

1.5 

t 

Behavior  management  (percent) 

88.5 

93.1 

-4.6 

0.472 

95.1 

89.5 

5.6 

0.463 

79.8 

89.4 

-9.6 

0.193 

See  notes  at  end  of  table. 


TJ 

> 


I 

CO 


Chapter  6.  Promoting  Alternative  Thinking  Strategies 


Table  6.13.  Impacts  on  teacher-reported  SACD  classroom  activities — PATHS — Continued 

Panel  2:  Engagement  in  any  activities  to  promote  SACD  goals  for  at  least  1 hour  per  week 

Year  1 Year  2 Year  3 

(Spring  3rd  grade) (Spring  4th  grade) (Spring  5th  grade) 

T reat-  T reat-  T reat- 


SACD  activity 

ment 

Control 

Impact 

p-value 

ment 

Control 

Impact 

p-value 

ment 

Control 

Impact 

p-value 

Teacher  sample  size 

60 

56 

55 

57 

50 

61 

Violence  prevention  and  peace  promotion 
(percent) 

48.0* 

15.9 

32.0 

0.020 

41.6 

22.5 

19.1 

0.121 

45.8 

33.2 

12.6 

0.343 

Social  and  emotional  development 
(percent) 

60.0* 

12.5 

47.5 

0.003 

57.1* 

21.3 

35.8 

0.023 

59.2* 

32.0 

27.2 

0.012 

Character  education  (percent) 

52.6* 

25.1 

27.5 

0.036 

63.9* 

25.5 

38.4 

0.001 

60. 2A 

37.1 

23.1 

0.078 

Tolerance  and  diversity  (percent) 

40.9* 

21.3 

19.6 

0.031 

43.3* 

16.2 

27.1 

0.043 

32.2 

25.9 

6.3 

0.596 

Risk  prevention  and  health  promotion 
(percent) 

32.8 

18.5 

14.3 

0.288 

34.5 

39.9 

-5.4 

0.699 

30.8 

27.2 

3.6 

0.778 

Civic  responsibility  and  community  service 
(percent) 

19.8 

10.0 

9.8 

0.374 

17.1 

10.9 

6.2 

0.239 

24.0* 

5.6 

18.4 

0.039 

Any  SACD  goal  (percent) 

74.7 

45.6 

29.1 

0.053 

86.7 

82.8 

3.9 

0.663 

82.4 

77.0 

5.4 

0.599 

Behavior  management  (percent) 

68.7 

72.7 

-4.0 

0.614 

82.2 

71.0 

11.2 

0.442 

58.9 

70.8 

-11.9 

0.305 

See  notes  at  end  of  table. 
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Table  6.13.  Impacts  on  teacher-reported  SACD  classroom  activities — PATHS — Continued 

Panel  3:  Engagement  in  activities  to  promote  SACD  goals  linked  to  named  SACD  programs2 

Year  1 Year  2 Year  3 

(Spring  3rd  grade) (Spring  4th  grade) (Spring  5th  grade) 

T reat-  T reat-  T reat- 


SACD  activity 

ment 

Control 

Impact 

p-value 

ment 

Control 

Impact 

p-value 

ment 

Control 

Impact 

p-value 

Teacher  sample  size 

60 

56 

55 

57 

50 

61 

Violence  prevention  and  peace  promotion 
(percent) 

54.4* 

13.8 

40.6 

0.009 

60.8* 

11.0 

49.8 

0.001 

65.7* 

30.2 

35.5 

0.034 

Social  and  emotional  development 
(percent) 

65.1* 

6.0 

59.1 

0.003 

78.1* 

14.2 

64.0 

0.000 

75.9* 

22.6 

53.3 

0.000 

Character  education  (percent) 

54.5* 

5.3 

49.2 

0.002 

70.2* 

5.1 

65.2 

0.000 

63.8* 

24.0 

39.8 

0.033 

Tolerance  and  diversity  (percent) 

t* 

t 

42.1 

0.006 

41.8* 

11.0 

30.8 

0.003 

42.5* 

8.4 

34.1 

0.016 

Risk  prevention  and  health  promotion 
(percent) 

25.1 

16.2 

8.9 

0.552 

23.3 

30.2 

-7.0 

0.569 

25.6 

30.4 

-4.8 

0.698 

Civic  responsibility  and  community  service 
(percent) 

6.6 

0.0 

6.6 

t 

t* 

t 

12.9 

0.035 

17.8 

5.6 

12.2 

0.229 

Any  named  activity  (percent) 

66.3* 

22.8 

43.5 

0.014 

83.3* 

45.9 

37.4 

0.003 

82.3* 

49.0 

33.3 

0.021 

See  notes  at  end  of  table. 
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Table  6.13.  Impacts  on  teacher-reported  SACD  classroom  activities — PATHS — Continued 


Panel  4:  Engagement  in  activities  to  promote  SACD  goals  linked  to  named  SACD  programs  for  at  least  1 hour  per  week 


Year  1 

(Spring  3rd  grade) 

Year  2 

(Spring  4th  grade) 

Year  3 

(Spring  5th  grade) 

SACD  activity 

Treat- 

ment 

Control 

Impact 

p-value 

T reat- 
ment 

Control 

Impact 

p-value 

T reat- 
ment 

Control 

Impact 

p-value 

Teacher  sample  size 

60 

56 

55 

57 

50 

61 

Violence  prevention  and  peace  promotion 
(percent) 

37. 5A 

11.7 

25.8 

0.100 

34. 5A 

7.7 

26.8 

0.062 

40. 8A 

17.2 

23.6 

0.094 

Social  and  emotional  development 
(percent) 

47.3 

0.0 

47.3 

t 

r 

t 

43.4 

0.003 

50.6* 

6.9 

43.7 

0.000 

Character  education  (percent) 

36.4 

0.0 

36.4 

t 

t* 

t 

43.4 

0.004 

52.4* 

11.7 

40.7 

0.015 

Tolerance  and  diversity  (percent) 

29.8 

0.0 

29.8 

t 

r 

% 

19.2 

0.016 

t* 

t 

24.2 

0.015 

Risk  prevention  and  health  promotion 
(percent) 

16.7 

7.8 

8.9 

0.499 

10.9 

21.7 

-10.7 

0.388 

18.9 

7.9 

11.0 

0.305 

Civic  responsibility  and  community  service 
(percent) 

5.5 

0.0 

5.5 

t 

4.6 

0.0 

4.6 

t 

12.2 

0.0 

12.2 

t 

Any  named  activity  (percent) 

52.2* 

10.2 

42.0 

0.040 

57. 3A 

28.4 

28.8 

0.060 

61.9* 

22.2 

39.6 

0.031 

t Not  applicable. 

J Reporting  standards  not  met.  Values  suppressed  to  protect  confidentiality. 

* Treatment  group  significantly  different  from  control  group  at  the  .05  level. 

A Treatment  group  significantly  different  from  control  group  at  the  .10  to  > .05  level. 

1 1n  Year  1 , the  omnibus  impact  for  all  the  outcomes  measured  together  was  positive  and  statistically  significant  on  the  basis  of  a multivariate  statistical  test.  In  Years  1 and  2,  at  least 
one  outcome  remained  positive  and  statistically  significant  and  no  outcome  was  negative  and  statistically  significant  after  applying  the  Benjamini-Hochberg  (1995)  procedure  to  adjust 
significance  levels  downward  to  account  for  the  multiple  testing  of  impacts. 

2 In  all  3 years  at  least  half  of  the  impacts  were  positive  and  statistically  significant  and  no  impact  was  negative  and  statistically  significant  on  the  basis  of  univariate  statistical  tests,  and 
at  least  one  outcome  remained  positive  and  statistically  significant  and  no  outcome  was  negative  and  statistically  significant  after  applying  the  Benjamini-Hochberg  (1995)  procedure  to 
adjust  significance  levels  downward  to  account  for  the  multiple  testing  of  impacts.  > 

NOTE:  Weights,  which  assign  equal  weight  to  each  school  within  the  program,  were  used  in  producing  the  treatment,  control,  and  overall  means.  H 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Use  of  Materials  and  Strategies 

For  use  of  materials  and  strategies  to  support  SACD  goals,  87  comparisons  were  made,  with  5 expected  to  be 
significant  by  chance.  Six  significant  impacts  were  found  on  treatment  teachers’  use  of  materials  and 
strategies.  In  Year  1,  more  treatment  teachers  used  role-playing  (impact  = 34  percentage  points).  In  Year  2, 
more  treatment  teachers  used  instructional  aids  (impact  = 30  percentage  points),  children’s  literature  (impact 
= 30  percentage  points),  role-playing  (impact  = 29  percentage  points),  and  direct  instruction  of  SACD 
(impact  = 20  percentage  points),  and  these  teachers  also  reported  using  a greater  average  number  of  strategies 
(by  1.6  strategies  on  average).  The  PATHS  impact  on  the  domain  of  materials  and  strategies  was  not 
statistically  significant  in  any  of  the  3 years. 
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Table  6.14.  Impacts  on  use  of  SACD  classroom  materials  and  teaching  strategies — PATHS 


Year  1 Year  2 Year  3 


(Spring  3rd  grade) 

(Spring  4th  grade) 

(Spring  5th  grade) 

SACD  material  and  teaching  strategy 

T reat- 
ment 

Control 

Impact  p-value 

T reat- 
ment 

Control 

Impact 

p-value 

Treat- 

ment 

Control 

Impact 

p-value 

Teacher  sample  size 

60 

56 

55 

57 

50 

61 

Use  of  SACD  materials  (percent) 

Teacher  guides  (manuals,  curricula) 

74.8 

64.7 

10.1 

0.409 

83.9 

56.9 

27.0 

0.116 

89.5 

79.0 

10.5 

0.239 

Student  materials  (workbooks  or  sheets) 

64.1 

66.6 

-2.5 

0.776 

72.0 

70.6 

1.4 

0.891 

77.4 

72.1 

5.3 

0.679 

Instructional  aids  (games,  software,  videos) 

50.0 

49.0 

1.0 

0.952 

61.9* 

32.4 

29.5 

0.022 

54. 2A 

37.2 

17.0 

0.088 

Giveaways  (bookmarks,  stickers) 

32. 6A 

57.8 

-25.2 

0.064 

41.9 

37.7 

4.2 

0.613 

35.4 

46.8 

-11.4 

0.344 

Children’s  literature 

73.3 

63.2 

10.1 

0.432 

75.2* 

45.5 

29.7 

0.012 

68.1 

61.9 

6.2 

0.471 

Other  types  of  materials 

10.4 

5.8 

4.5 

0.468 

5.2 

7.0 

-1.9 

0.755 

t 

t 

11.6 

0.19 

Did  not  use  any  of  these  materials 

9.3 

7.5 

1.8 

0.765 

t 

t 

-9.2 

0.125 

t 

t 

-5.7 

0.254 

Use  of  teaching  strategies  (percent) 

Role-playing 

91.6* 

58.1 

33.5 

0.024 

90.4* 

61.0 

29.4 

0.002 

92.2 

84.6 

7.6 

0.282 

Cooperative  learning 

98.8 

100.0 

-1.3 

t 

100.0 

100.0 

0.0 

t 

100.0 

100.0 

0.0 

T 

Peer  group  discussions 

92.1 

89.5 

2.6 

0.590 

100.0 

91.4 

8.6 

t 

93.5 

100.0 

-6.5 

t 

Direct  instruction  of  SACD 

87.9 

85.0 

2.8 

0.757 

98.5* 

79.0 

19.5 

0.021 

100.0 

98.1 

1.9 

t 

Skill  training 

54.8 

45.3 

9.5 

0.324 

67. 9A 

44.1 

23.8 

0.058 

87.8 

81.0 

6.8 

0.360 

Incorporating  SACD  into  academic 
curriculum 

84.5 

80.6 

3.9 

0.572 

89. 5A 

77.2 

12.3 

0.072 

95.9 

92.4 

3.4 

0.413 

Parent  training 

5.6 

9.6 

-4.1 

0.388 

7.7 

8.4 

-0.8 

0.902 

25.7 

25.2 

0.5 

0.963 

Parent/community  involvement 

20.7 

38.4 

-17.7 

0.212 

19.5 

37.5 

-18.0 

0.138 

40.3 

36.6 

3.7 

0.725 

Mentoring 

29.0 

40.4 

-11.5 

0.289 

24.8 

24.0 

0.8 

0.932 

54.1 

54.4 

-0.3 

0.961 

Good  behavior  notes  sent  home  daily 
or  weekly 

82.0 

73.9 

8.1 

0.344 

81.3 

74.5 

6.9 

0.642 

93.1 

98.1 

-5.1 

0.268 

Presenting  role  models 

86.6 

81.2 

5.4 

0.363 

75.7 

71.6 

4.0 

0.663 

94.1 

87.8 

6.3 

0.184 

See  notes  at  end  of  table. 
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Table  6.14.  Impacts  on  use  of  SACD  classroom  materials  and  teaching  strategies — PATHS — Continued 


Year  1 Year  2 Year  3 


(Spring  3rd  grade) 

(Spring  4th  grade) 

(Spring  5th  grade) 

SACD  material  and  teaching  strategy 

T reat- 
ment 

Control 

Impact 

p-value 

T reat- 
ment 

Control 

Impact 

p-value 

Treat- 

ment 

Control 

Impact 

p-value 

Use  of  teaching  strategies  (percent) — 
Continued 

Targeted  story  reading  or  writing  on  social 
and  character  development  themes 

88.9 

84.0 

4.9 

0.599 

90.8 

84.6 

6.3 

0.173 

98.2 

93.2 

5.0 

0.212 

Peer  mediation 

50.4 

49.1 

1.3 

0.935 

59.1 

46.0 

13.1 

0.523 

67.3 

63.7 

3.6 

0.749 

Honor  roll  for  positive  behavior 

40.7 

40.1 

0.5 

0.974 

45.3 

28.3 

17.0 

0.253 

69.3 

55.4 

13.9 

0.341 

Pledges  or  recitations  on  social  and 
character  development  themes 

21.8 

56.7 

-34.9 

0.173 

47.6 

55.7 

-8.1 

0.722 

54.5 

75.6 

-21.0 

0.265 

Guided  visualization 

58.6 

52.4 

6.2 

0.500 

67.3 

46.8 

20.4 

0.116 

62.6 

56.0 

6.6 

0.447 

Student-led/student-assisted  instruction 

54.9 

54.4 

0.5 

0.952 

70.3 

57.0 

13.4 

0.185 

78.5 

70.0 

8.5 

0.377 

Journaling 

75.9 

77.9 

-1.9 

0.853 

85.7 

79.2 

6.5 

0.261 

84.0 

85.5 

-1.5 

0.778 

Time  out  for  negative  behavior 

86.9 

73.5 

13.4 

0.239 

82.9 

75.4 

7.6 

0.479 

89.5 

80.2 

9.4 

0.410 

Daily  or  weekly  rewards  for  positive 
behavior 

92.2 

91.3 

0.9 

0.889 

86.9 

95.7 

-8.8 

0.273 

96.6 

96.7 

-0.1 

0.990 

Any  strategy 

100.0 

100.0 

0.0 

t 

100.0 

100.0 

0.0 

t 

100.0 

100.0 

0.0 

t 

Number  of  strategies  (mean) 

12.9 

12.7 

0.2 

0.715 

13.9* 

12.3 

1.6 

0.039 

15.6 

15.2 

0.4 

0.477 

f Not  applicable. 

$ Reporting  standards  not  met.  Values  suppressed  to  protect  confidentiality. 

* Treatment  group  significantly  different  from  control  group  at  the  .05  level. 

A Treatment  group  significantly  different  from  control  group  at  the  .10  to  > .05  level. 

NOTE:  Weights,  which  assign  equal  weight  to  each  school  within  the  program,  were  used  in  producing  the  treatment,  control,  and  overall  means. 
SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Regarding  the  use  of  schoolwide  strategies,  18  comparisons  were  made  between  treatment  and  control 
teacher  reports,  with  1 expected  to  be  significant  by  chance.  There  was  1 statistically  significant  difference 
between  reports  from  treatment  and  control  teachers  on  their  use  of  schoolwide  strategies  in  Year  1 (these 
data  are  not  shown  in  a table),  with  fewer  treatment  teachers  reporting  use  of  special  events  than  control 
teachers  (impact  = -17  percentage  points).  The  overall  impact  of  the  PATHS  program  on  the  domain  for  use 
of  schoolwide  strategies  in  named  SACD  activities  was  not  significant  in  any  year. 

Participation  in  Professional  Development 

Regarding  reported  participation  in  professional  development,  27  comparisons  were  made  over  3 years,  with 
2 expected  to  be  significant  by  chance.  In  Year  1,  the  intervention  had  a statistically  significant  effect  on 
treatment  teachers’  participation  in  professional  development,  with  more  treatment  teachers  than  control 
teachers  reporting  SACD  training  in  the  past  12  months  (impact  = 32  percentage  points)  and  reporting  more 
hours  of  training  (by  10.6  hours  on  average).  In  terms  of  specific  SACD  goals,  more  treatment  teachers 
reported  training  in  violence  prevention  and  peace  promotion  (impact  = 24  percentage  points),  social  and 
emotional  development  (impact  = 47  percentage  points),  and  character  education  (impact  = 46  percentage 
points).  In  Year  2,  PATHS  had  an  impact  on  social  and  emotional  development  training  (impact  = 17 
percentage  points).  There  was  a significant  impact  on  the  domain  in  Year  1. 
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Table  6.15.  Impacts  on  teacher-reported  SACD  professional  development — PATHS 


Year  1 Year  2 Year  3 

(Spring  3rd  grade) (Spring  4th  grade) (Spring  5th  grade) 

T reat-  T reat-  T reat- 


SACD  professional  development 

ment 

Control 

Impact 

p-value 

ment 

Control 

Impact 

p-value 

ment 

Control 

Impact 

p-value 

Teacher  sample  size 

60 

56 

55 

57 

50 

61 

SACD  training  in  past  12  months  (percent) 

91.9* 

60.1 

31.8 

0.020 

76.4 

66.9 

9.5 

0.294 

63.8 

70.0 

-6.2 

0.461 

Hours  of  SACD  training  (mean) 

14.0* 

3.4 

10.6 

0.000 

5.2 

4.4 

0.7 

0.465 

7.1 

5.5 

1.6 

0.516 

Training  by  goal  (percent) 

Violence  prevention  and  peace  promotion 

44.8* 

20.6 

24.2 

0.034 

18.7 

14.6 

4.0 

0.737 

20.3 

11.5 

8.8 

0.432 

Social  and  emotional  development 

74.2* 

27.5 

46.7 

0.005 

37.6* 

20.8 

16.9 

0.025 

29.8 

28.2 

1.6 

0.926 

Character  education 

67.1* 

21.2 

45.9 

0.002 

45. 7A 

25.7 

20.0 

0.056 

33.9 

16.1 

17.8 

0.112 

Tolerance  and  diversity 

16.3 

17.0 

-0.7 

0.945 

31.4 

41.6 

-10.2 

0.257 

41.3 

40.1 

1.2 

0.937 

Risk  prevention  and  health  promotion 

15.5 

23.0 

-7.5 

0.524 

15.5 

9.4 

6.1 

0.536 

8.5 

18.1 

-9.6 

0.231 

Civic  responsibility  and  community  service 

6.5 

5.4 

1.1 

0.829 

6.1 

7.1 

-1.0 

0.811 

t 

i 

-1.6 

0.727 

Behavior  management 

40.5 

32.7 

7.8 

0.697 

30.6 

27.7 

2.9 

0.842 

22.5 

30.7 

-8.2 

0.554 

4:  Reporting  standards  not  met.  Values  suppressed  to  protect  confidentiality. 

* Treatment  group  significantly  different  from  control  group  at  the  .05  level. 

A T reatment  group  significantly  different  from  control  group  at  the  . 1 0 to  > .05  level. 

1 1n  Year  1 , at  least  half  of  the  impacts  were  positive  and  statistically  significant  and  no  impact  was  negative  and  statistically  significant  on  the  basis  of  univariate  statistical  tests,  and  at 
least  one  outcome  remained  positive  and  statistically  significant  and  no  outcome  was  negative  and  statistically  significant  after  applying  the  Benjamini-Hochberg  (1995)  procedure  to 
adjust  significance  levels  downward  to  account  for  the  multiple  testing  of  impacts. 

NOTE:  Weights,  which  assign  equal  weight  to  each  school  within  the  program,  were  used  in  producing  the  treatment,  control,  and  overall  means. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Chapter  6.  Promoting  Alternative  Thinking  Strategies  (PATHS) 


Attitudes  and  Practices 

Teachers  reported  on  their  enthusiasm  for  SACD  efforts  in  their  schools  (these  data  are  not  shown  in  a table) 
by  indicating  enthusiasm,  cooperation,  or  open  dislike.  They  also  reported  on  the  SACD  practices  of  teachers 
and  staff  members  in  their  schools  (these  data  are  not  shown  in  a table).  These  practices  included  modeling 
positive  character  and  behavior  traits  with  students  and  fellow  teachers,  involving  students  in  making 
decisions,  giving  students  a voice  in  school  governance,  the  school  encouraging  parent  involvement  in 
children’s  social  and  character  development,  and  using  developmentally  appropriate  discipline  strategies 
rather  than  punishment  for  misbehavior.  Twenty-seven  comparisons  were  made  over  3 years,  with  1 expected 
to  be  significant  by  chance.  There  were  no  statistically  significant  impacts  on  teachers’  enthusiasm  for  SACD 
efforts  in  their  schools  in  any  of  the  years,  nor  were  there  significant  differences  in  the  treatment  and  control 
teachers’  reports  of  the  overall  use  of  practices  conducive  to  students’  social  and  character  development. 
PATHS  had  no  impact  on  the  domain  in  any  year. 

Year-by-Year  Impacts  on  Students  and  Perceptions  of  School  Climate 

The  primary  research  question  for  the  PATHS  evaluation  was  this: 

What  is  the  average  effect  of  the  PATHS  program  on  children’s  social  and  emotional  competence, 
behavior,  academic  achievement,  and  perceptions  of  school  climate? 

The  first  approach  to  answering  this  question  was  to  examine  the  year -by-year  impacts  of  PATHS  on  these 
student  and  school  climate  outcomes  over  the  3 years  as  the  students  progressed  from  third  through  fifth 
grades. 

Equation  (2)  (described  in  chapter  1)  was  estimated  to  provide  the  impacts  of  the  PATHS  program  on  the  20 
outcome  measures  using  data  from  the  10  treatment  and  control  schools.  For  the  PATHS  evaluation, 
equation  (2)  excluded  the  program  fixed  effects  (6p)  and  included  program-specific  covariates  and  random 
school  effects  covariates.  Table  6.16  lists  the  covariates  used  with  outcomes  from  each  report  in  the  PATHS 
analysis. 
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Table  6.16.  Covariates  used  with  outcomes  from  each  report  for  analysis — PATHS 


Potential  covariate 

CR 

outcome 

PCR 

outcome 

TRS 

outcome 

TRCS 

outcome 

Total  number 

18 

27 

27 

5 

Child-reported 

Female 

✓ 

✓ 

✓ 

Hispanic 

✓ 

✓ 

✓ 

Black  (non-Hispanic) 

✓ 

V 

✓ 

Other  ethnicity 

✓ 

V 

✓ 

Age  in  years 

✓ 

V 

✓ 

Scales 

Afraid  at  School 

✓ 

Altruistic  Behavior 

Empathy 

Engagement  with  Learning 

Negative  School  Orientation 

V 

✓ 

Normative  Beliefs  About  Aggression 

Sense  of  School  as  a Community 

Problem  Behavior 

V 

Self-Efficacy  for  Peer  Interactions 

✓ 

V 

✓ 

Victimization  at  School 

✓ 

Primary  caregiver-reported 

Age  in  years 

Completed  high  school  or  equivalent 

Some  college 

Bachelor’s  or  higher  degree 

Highest  level  of  education  in  household 

Completed  high  school  or  equivalent 

✓ 

V 

✓ 

Some  college 

✓ 

V 

✓ 

Bachelor’s  or  higher  degree 

✓ 

V 

✓ 

Mother  present  in  home  life 

V 

✓ 

Mother  and  father  present 

V 

Respondent  someone  other  than  mother  or  father 

V 

Number  of  people  in  household 

Household  Income:  $20,000  to  $40,000 

✓ 

V 

✓ 

Household  Income:  $40,000  to  $60,000 

✓ 

V 

✓ 

Household  Income:  More  than  $60,000 

✓ 

V 

✓ 

Income-to-poverty-threshold  ratio:  Below  135  percent 

V 

Income-to-poverty-threshold  ratio:  135-185  percent 

V 

Full-time  employment 

V 

✓ 

Part-time  employment 

✓ 

V 

✓ 

See  notes  at  end  of  table. 
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Table  6.16.  Covariates  used  with  outcomes  from  each  report  for  analysis — PATHS — Continued 


Potential  covariate 

CR 

outcome 

PCR 

outcome 

TRS 

outcome 

TRCS 

outcome 

Parental  scales 

APQ-Poor  Monitoring  and  Supervision  Subscale 

✓ 

✓ 

✓ 

APQ-Positive  Parenting  Subscale 

Child-Centered  Social  Control 

✓ 

Confusion,  Hubbub,  and  Order 

Community  Resources 

✓ 

Community  Risk 

✓ 

✓ 

Parent  and  Teacher  Involvement 

s 

✓ 

Child  scales 

Altruistic  Behavior 

Positive  Social  Behavior 

✓ 

Problem  Behavior 

V 

Teacher-reported 

Female 

✓ 

Hispanic 

✓ 

Black  (non-Hispanic) 

✓ 

Other  ethnicity 

✓ 

Total  teaching  experience 

✓ 

Total  experience  in  current  school 

Regular  certificate 

Other  certificate 

Highest  degree-bachelor’s 

Child  scales 

Academic  Competence  and  Motivation 

ADHD-Related  Behavior 

✓ 

✓ 

Altruistic  Behavior 

s 

Positive  Social  Behavior 

Problem  Behavior 

V 

V 

Parent  and  Teacher  Involvement 

✓ 

V 

V 

NOTES:  Abbreviations  are 
CR:  Child  Report 
PCR:  Primary  Caregiver  Report 
TRS:  Teacher  Report  on  Student 
TRCS:  Teacher  Report  on  Classroom  and  School 
ADHD:  Attention  deficit  hyperactivity  disorder 
APC:  Alabama  Parenting  Cuestionnaire 
v' : Covariate  used 
Blank  cell:  Covariate  not  used 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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To  assess  the  statistical  power  of  the  program -level  impact  estimates,  minimum  detectable  impacts  in  effect 
size  units  (MDES)  for  each  outcome  measure  were  calculated  for  the  PATHS  evaluation  (table  6.17).  MDES 
represent  the  smallest  impacts  in  effect  size  (standard  deviation)  units  that  can  be  detected  with  a high 
probability  (80%).  MDES  are  primarily  a function  of  study  sample  sizes,  the  degrees  of  freedom  available  for 
statistical  tests,  and  design  effects  due  to  clustering  (Schochet  2005).  For  the  PATHS  evaluation,  the  MDES 
ranged  from  0.137  to  0.709  for  the  child-level  outcomes  based  on  the  Child,  Primary  Caregiver,  and  Teacher 
Report  on  Student  and  from  0.652  to  1.040  for  the  school  climate  outcomes  based  on  the  Teacher  Report  on 
Classroom  and  School.  In  general,  the  MDES  for  the  school  climate  outcomes  were  larger  than  those  for  the 
child-level  outcomes. 
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Table  6.17.  Adjusted  minimum  detectable  effect  sizes  for  impact  evaluation — PATHS 


Outcome  measure-Report 

Year  1 

Year  2 

Year  3 

Social  and  Emotional  Competence  Domain 

Self-Efficacy  for  Peer  Interaction-CR 

0.160 

0.210 

0.179 

Normative  Beliefs  About  Aggression-CR 

0.137 

0.201 

0.209 

Empathy-CR 

0.252 

0.171 

0.239 

Behavior  Domain 

Altruistic  Behavior-CR 

0.230 

0.159 

0.228 

Altruistic  Behavior-PCR 

0.239 

0.154 

0.210 

Altruistic  Behavior-TRS 

0.423 

0.282 

0.573 

Positive  Social  Behavior-PCR 

0.147 

0.218 

0.156 

Positive  Social  Behavior-TRS 

0.188 

0.467 

0.395 

Problem  Behavior-CR 

0.137 

0.138 

0.213 

Problem  Behavior-PCR 

0.194 

0.173 

0.156 

Problem  Behavior-TRS 

0.181 

0.419 

0.230 

ADHD-Related  Behavior-TRS 

0.172 

0.440 

0.230 

Academics  Domain 

Engagement  with  Learning-CR 

0.140 

0.178 

0.138 

Academic  Competence  and  Motivation-TRS 

0.217 

0.169 

0.312 

Perceptions  of  School  Climate  Domain 

Positive  School  Orientation-CR 

0.196 

0.611 

0.709 

Negative  School  Orientation-CR 

0.256 

0.332 

0.246 

Student  Afraid  at  School-CR 

0.174 

0.142 

0.249 

Victimization  at  School-CR 

0.164 

0.151 

0.252 

Feelings  of  Safety-TRCS 

0.652 

1.040 

1.034 

Student  Support  for  Teachers-TRCS 

0.792 

0.937 

0.913 

NOTE:  Abbreviations  are 
CR:  Child  Report 
PCR:  Primary  Caregiver  Report 
TRS:  Teacher  Report  on  Student 
TRCS:  Teacher  Report  on  Classroom  and  School 
ADHD:  Attention  deficit  hyperactivity  disorder 
The  minimum  detectable  effect  (MDE)  formula  used  in  the  calculations  is  as  follows: 


MDE  = f actor (df)  * 


Pi 


(-  + -)  + (l-Pi)  (— + — ) 
\sT  scJ  \srnr  srnr) 


sTnT  scnc) 


where  sT  and  sc  are  the  number  of  treatment  and  comparison  schools;  nT  and  nc  are  the  average  number  of  students  per 
classroom;  p,  is  the  intraclass  correlation  (ICC)  at  the  school  level;  and  factor(df)  is  a constant  that  depends  on  the  number  of 
degrees  of  freedom  (df)  available  for  analysis  (and  is  2.802  for  the  pooled  analysis). 


SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Table  6.18  provides  the  estimates  of  the  PATHS  program’s  impacts  on  each  of  the  20  outcome  measures 
over  each  of  the  3 years  (60  impacts  in  total,  with  3 expected  to  be  statistically  significant  by  chance).  Of  the 
60  results,  none  were  statistically  significant.  In  Year  3,  substantively  important  but  nonsignificant  detrimental 
impacts  were  found  for  Altruistic  Behavior  (Teacher  Report  on  Student,  effect  size  [ES]  = -0.31)  and  Feelings 
of  Safety  (Teacher  Report  on  Classroom  and  School,  ES  = -0.29).  Application  of  the  heuristics  to  adjust  for 
multiple  comparisons  within  each  outcome  domain  indicates  a statistically  significant  detrimental  effect  on 
the  Social  and  Emotional  Competence  domain  in  Year  2 and  a statistically  significant  detrimental  effect  on 
the  Academics  domain  in  Year  3. 
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Table  6.18.  Impacts  on  child  and  school  outcomes — PATHS 


Year  1 

(Spring  3rd  grade) 

Year  2 

(Spring  4th  grade) 

Year  3 

(Spring  5th  grade) 

Scale-Report 

T reat- 
ment 

Control 

Effect 

size 

p-value 

T reat- 
ment 

Control 

Effect 

size 

p-value 

T reat- 
ment 

Control 

Effect 

size 

p-value 

Social  and  Emotional  Competence  Domain1 

Self-Efficacy  for  Peer  Interactions-CR  (+) 

3.06 

3.06 

0.01 

0.953 

3.18 

3.28 

-0.16 

0.261 

3.21 

3.30 

-0.16 

0.238 

Normative  Beliefs  About  Aggression-CR  (-) 

1.21 

1.23 

-0.05 

0.606 

1.26 

1.37 

-0.17 

0.214 

1.41 

1.42 

-0.01 

0.951 

Empathy-CR  (+) 

2.38 

2.34 

0.10 

0.493 

2.21 

2.14 

0.13 

0.245 

2.09 

2.11 

-0.03 

0.851 

Behavior  Domain 

Altruistic  Behavior-CR  (+) 

1.25 

1.13 

0.15 

0.320 

1.00 

0.96 

0.06 

0.635 

0.98 

0.98 

0.00 

0.991 

Altruistic  Behavior-PCR  (+) 

2.15 

2.13 

0.03 

0.827 

2.06 

2.19 

-0.16 

0.140 

2.12 

2.16 

-0.06 

0.665 

Altruistic  Behavior-TRS  (+) 

1.26 

1.25 

0.02 

0.932 

1.22 

1.18 

0.17 

0.395 

1.27 

1.37 

-0.31 0 

0.485 

Positive  Social  Behavior-PCR  (+) 

2.96 

3.03 

-0.13 

0.159 

3.05 

3.06 

-0.01 

0.953 

3.04 

3.09 

-0.08 

0.413 

Positive  Social  Behavior-TRS  (+) 

3.01 

2.98 

0.05 

0.590 

3.14 

3.04 

0.15 

0.503 

3.18 

3.14 

0.06 

0.778 

Problem  Behavior-CR  (-) 

0.22 

0.27 

-0.10 

0.220 

0.27 

0.30 

-0.06 

0.547 

0.43 

0.37 

0.12 

0.385 

Problem  Behavior-PCR  (-) 

1.57 

1.54 

0.10 

0.383 

1.53 

1.53 

0.00 

0.987 

1.53 

1.55 

-0.04 

0.714 

Problem  Behavior-TRS  (-) 

1.44 

1.42 

0.03 

0.677 

1.42 

1.42 

0.01 

0.957 

1.36 

1.43 

-0.14 

0.229 

ADHD-Related  Behavior-TRS  (-) 

1.75 

1.79 

-0.07 

0.417 

1.62 

1.70 

-0.13 

0.574 

1.55 

1.69 

-0.22 

0.126 

Academics  Domain2 

Engagement  with  Learning-CR  (+) 

3.73 

3.70 

0.05 

0.657 

3.70 

3.69 

0.01 

0.954 

3.61 

3.66 

-0.11 

0.296 

Academic  Competence  and  Motivation- 

TRS  (+) 

3.13 

3.11 

0.02 

0.831 

2.95 

3.05 

-0.09 

0.272 

3.11 

3.03 

0.08 

0.621 

See  notes  at  end  of  table.  _ 
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Table  6.18.  Impacts  on  child  and  school  outcomes — PATHS — Continued 


Year  1 

(Spring  3rd  grade) 

Year  2 

(Spring  4th  grade) 

Year  3 

(Spring  5th  grade) 

Scale-Report 

Treat- 

ment 

Control 

Effect 
size  i 

o-value 

T reat- 
ment 

Control 

Effect 

size 

p-value 

T reat- 
ment 

Control 

Effect 

size 

p-value 

Perceptions  of  School  Climate  Domain 
Positive  School  Orientation-CR  (+) 

2.96 

2.98 

-0.03 

0.826 

2.79 

2.67 

0.15 

0.640 

2.66 

2.66 

0.00 

0.997 

Negative  School  Orientation-CR  (-) 

1.76 

1.81 

-0.08 

0.587 

1.87 

1.90 

-0.05 

0.765 

1.93 

2.04 

-0.19 

0.214 

Student  Afraid  at  School-CR  (-) 

2.17 

2.18 

-0.01 

0.945 

2.08 

2.09 

-0.01 

0.909 

2.09 

2.28 

-0.22 

0.180 

Victimization  at  School-CR  (-) 

0.68 

0.65 

0.04 

0.681 

0.65 

0.56 

0.12 

0.306 

0.71 

0.57 

0.20 

0.284 

Feelings  of  Safety-TRCS  (+) 

3.28 

3.30 

-0.02 

0.932 

3.36 

3.18 

0.20 

0.615 

3.19 

3.40 

-0.29° 

0.582 

Student  Support  for  Teachers-TRCS  (+) 

3.54 

3.53 

0.01 

0.980 

3.52 

3.46 

0.07 

0.838 

3.57 

3.70 

-0.20 

0.631 

a Substantive  (but  nonsignificant  at  .05  level)  effect  size  of  s .25  or  < -.25. 

1 1mpact  on  domain  found  statistically  significant  and  detrimental  in  Year  2 based  on  the  fourth  heuristic  in  which  the  statistical  model  used  to  estimate  impacts  on  the  individual 
outcomes  was  re-estimated  using  a composite  of  all  the  outcome  variables  under  a domain.  The  domain  was  found  significant  if  the  impact  on  the  composite  was  significant.  The 
composite  was  formed  by  standardizing  each  outcome  variable  using  its  standard  deviation,  combining  the  values  of  the  outcome  variables,  and  taking  the  average  of  the  final  value. 

2 Impact  on  domain  found  statistically  significant  and  detrimental  in  Year  3 based  on  the  fourth  heuristic  in  which  the  statistical  model  used  to  estimate  impacts  on  the  individual 
outcomes  was  re-estimated  using  a composite  of  all  the  outcome  variables  under  a domain.  The  domain  was  found  significant  if  the  impact  on  the  composite  was  significant.  The 
composite  was  formed  by  standardizing  each  outcome  variable  using  its  standard  deviation,  combining  the  values  of  the  outcome  variables,  and  taking  the  average  of  the  final  value. 
NOTE:  Abbreviations  are 
CR:  Child  Report 
PCR:  Primary  Caregiver  Report 
TRS:  Teacher  Report  on  Student 
TRCS:  Teacher  Report  on  Classroom  and  School 
ADHD:  Attention  deficit  hyperactivity  disorder 

No  findings  were  found  statistically  significant  at  or  below  the  .05  level.  The  +/-  signs  in  parentheses  indicate  the  direction  of  a beneficial  outcome.  All  impact  estimates  were  calculated 
using  regression  models  where  each  program  and  school  within  a program  was  weighted  equally.  The  standard  errors  of  all  estimates  account  for  design  effects  due  to  unequal 
weighting  and  the  clustering  of  students  within  schools.  See  table  1 .5  for  information  about  the  measured  used  to  create  the  outcome  variables.  The  effect  size  was  calculated  by 
dividing  the  estimated  impact  by  the  standard  deviation  of  the  outcome  measure  for  the  control  group. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Impacts  on  Child  Outcomes  Over  Time 

The  PATHS  program’s  impacts  on  the  child  outcomes  over  time  were  estimated  using  growth  curve  models 
by  examining  treatment  and  control  group  differences  in  the  trajectories  of  student  outcomes  during  the 
follow-up  period  while  accounting  for  clustering  at  the  school  level.  The  growth  curve  models  are  estimated 
using  a three-level  hierarchical  linear  model,  where  Level  1 corresponds  to  time,  Level  2 to  students,  and 
Level  3 to  schools. 

Table  6.19  provides  the  estimates  of  the  PATHS  program’s  impacts  on  the  growth  in  student  outcome 
measures  over  the  3 years.  The  estimated  impacts  ranged  in  effect  size  units  (absolute  value)  from  0.01  to 
0.12.  One  of  the  18  estimated  PATHS  intervention  impacts  on  the  trajectories  of  child  outcomes  was 
statistically  significant  (no  more  than  expected  by  chance);  this  was  Academic  Competence  and  Motivation 
(Teacher  Report  on  Student,  ES  = 0.08). 


387 


388 


Table  6.19.  Impacts  on  growth  of  child  outcomes — PATHS 


Average  growth  in  the  score  per  year1 

Standard 


Scale-Report 

Mean  score  at 
implementation2 

Treatment 

Control 

Impact  on 
growth3 

Effect 

4 

size 

error  of 
impact 

p-value  of 
impact 

Social  and  Emotional  Competence  Domain 

Self-Efficacy  for  Peer  Interactions-CR  (+) 

2.94 

0.13 

0.14 

-0.01 

-0.02 

0.03 

0.693 

Normative  Beliefs  About  Aggression-CR  (-) 

1.17 

0.11 

0.08 

0.03 

0.07 

0.04 

0.512 

Empathy-CR  (+) 

2.44 

-0.18 

-0.13 

-0.05 

-0.12 

0.04 

0.179 

Behavior  Domain 

Altruistic  Behavior-CR  (+) 

1.35 

-0.22 

-0.13 

-0.09 

-0.07 

0.06 

0.187 

Altruistic  Behavior-PCR  (+) 

2.24 

-0.05 

-0.05 

-0.01 

-0.01 

0.03 

0.846 

Altruistic  Behavior-TRS  (+) 

1.19 

0.03 

0.05 

-0.02 

-0.05 

0.07 

0.803 

Positive  Social  Behavior-PCR  (+) 

2.96 

0.05 

0.04 

0.01 

0.01 

0.02 

0.698 

Positive  Social  Behavior-TRS  (+) 

2.99 

0.07 

0.04 

0.03 

0.04 

0.07 

0.642 

Problem  Behavior-CR  (-) 

0.17 

0.09 

0.08 

0.01 

0.02 

0.04 

0.812 

Problem  Behavior-PCR  (-) 

1.60 

-0.03 

-0.02 

-0.01 

-0.01 

0.01 

0.672 

Problem  Behavior-TRS  (-) 

1.38 

0.00 

0.01 

0.00 

-0.01 

0.02 

0.852 

ADHD-Related  Behavior-TRS  (-) 

1.81 

-0.09 

-0.06 

-0.03 

-0.03 

0.04 

0.535 

Academics  Domain 

Engagement  with  Learning-CR  (+) 

3.76 

-0.06 

-0.04 

-0.02 

-0.04 

0.03 

0.422 

Academic  Competence  and  Motivation-TRS  (+) 

2.93 

0.09* 

-0.02 

0.11 

0.08 

0.05 

0.048 

See  notes  at  end  of  table. 
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Table  6.19.  Impacts  on  growth  of  child  outcomes — PATHS — Continued 


Average  growth  in  the  score  per  year1 


Scale-Report 

Mean  score  at 
implementation2 

Treatment 

Control 

Impact  on 
growth3 

Effect 

size4 

Standard 
error  of 
impact 

p-value  of 
impact 

Perceptions  of  School  Climate  Domain 
Positive  School  Orientation-CR  (+) 

3.24 

-0.25 

-0.22 

-0.04 

-0.05 

0.06 

0.552 

Negative  School  Orientation-CR  (-) 

1.70 

0.13 

0.08 

0.05 

0.06 

0.05 

0.258 

Student  Afraid  at  School-CR  (-) 

2.31 

-0.07 

-0.08 

0.01 

0.01 

0.05 

0.790 

Victimization  at  School-CR  (-) 

0.69 

-0.04 

-0.02 

-0.02 

-0.01 

0.04 

0.719 

* Treatment  group  significantly  different  from  control  group  at  the  .05  level. 

1 Pertains  to  the  estimated  slope  of  the  outcome  for  the  treatment  or  control  groups. 

2 The  average  score  at  implementation  is  calculated  across  treatment  and  control  groups,  using  regression  models  for  adjustment  on  covariates. 

3 Estimated  difference  between  the  slope  of  the  treatment  and  control  groups. 

4 Effect  size:  the  slope  of  the  treatment  group  minus  the  slope  of  the  control  group  divided  by  the  standard  deviation  of  the  outcome  for  the  program's  control  group  (the  standard 
deviation  is  calculated  without  accounting  for  school-level  clustering  or  regression  adjustments). 

NOTE:  Abbreviations  are 
CR:  Child  Report 
PCR:  Primary  Caregiver  Report 
TRS:  Teacher  Report  on  Student 
ADHD:  Attention  deficit  hyperactivity  disorder 

The  +/-  signs  in  parentheses  indicate  the  direction  of  a beneficial  outcome.  All  impact  estimates  were  calculated  using  HLM  6.06.  Sample  weights  were  used  in  all  analyses  to  give  (1) 
each  school  equal  weight  in  each  program  (within  each  time  period)  and  (2)  each  time  period  equal  weight  within  the  analysis.  See  table  1 .5  for  information  about  the  measures  used 
to  create  the  outcome  variables. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Chapter  6.  Promoting  Alternative  Thinking  Strategies  (PATHS) 


Summary 

As  part  of  the  Social  and  Character  Development  (SACD)  initiative,  researchers  at  the  New  York/Minnesota 
site  implemented  and  evaluated  the  PATHS  program.  This  program  focused  on  promoting  children’s  self- 
control,  emotional  understanding,  positive  self-esteem,  relationships,  and  interpersonal  problem-solving  skills. 
Ten  public  schools,  eight  in  New  York  and  two  in  Minnesota,  were  recruited  by  the  Children’s  Institute 
research  team  and  randomly  assigned  to  treatment  and  control  conditions  to  determine  the  impact  of  the 
PATHS  program  on  social  and  character  development  activities  in  the  schools  and  on  the  child  outcome 
domains  of  Social  and  Emotional  Competence,  Behavior,  Academics,  and  Perceptions  of  School  Climate. 

Analyses  of  the  initial  characteristics  of  the  sample  (students,  caregivers,  communities,  teachers,  and  schools) 
indicated  that  randomization  to  treatment  and  control  status  produced  groups  that  were  relatively  similar  at 
the  start  of  the  study  (with  2 out  of  84  comparisons  statistically  significantly  different,  fewer  than  would  be 
expected  by  chance).  The  data  on  the  initial  level  of  SACD  activity  led  to  two  findings.  First,  treatment 
teachers  reported  greater  use  of  and  training  in  SACD  activities  than  control  teachers,  and  they  did  so  more 
often  than  would  be  expected  by  chance  (8  out  of  62  comparisons,  with  3 expected  significant  by  chance). 
There  are  two  potential  causes  for  this  finding,  and  the  analysis  cannot  be  used  to  determine  whether  the 
reason  for  such  a difference  was  that  the  two  groups  did  differ  on  their  initial  use  of  SACD  activities  (i.e.,  that 
randomization  did  not  create  similar  treatment  and  control  groups)  or  the  fact  that  the  training  of  all 
treatment  teachers  and  the  implementation  of  the  PATHS  program  began  before  the  initial  data  were 
collected  (by  6 weeks)  influenced  the  teacher  reports.  Because  it  is  likely  (though  unproven)  that  the  training 
and  implementation  affected  the  teacher  reports,  these  data  were  not  considered  appropriate  for  use  as  a 
baseline  measure  of  SACD  activities  and  training  in  the  treatment  schools. 

Second,  these  data  indicate  that  the  control  condition  for  the  SACD  project  was  not  a “no  treatment”  control 
but  a “standard  practice”  control.  Because  the  control  teachers  were  not  affected  by  the  implementation  of 
the  SACD  programs  before  data  collection,  their  reports  reflected  standard  practice  in  the  control  schools. 
Standard  practice  at  the  control  schools  included  reports  of  60  percent  to  85  percent  of  teachers  using  any 
SACD  activities,  88  percent  of  teachers  using  specific  materials  in  conjunction  with  these  activities,  100 
percent  of  teachers  using  at  least  one  of  the  specified  instructional  strategies,  and  68  percent  of  teachers 
participating  in  SACD  training  over  the  past  12  months. 

Analyses  of  the  impacts  of  the  PATHS  program  on  the  level  of  SACD  activities  in  the  schools  revealed 
impacts  on  the  use  of  such  activities  (39  out  of  90)  and  related  materials  and  strategies  (7  out  of  87,  1 
negative)  across  the  3 years,  and  use  of  more  professional  development  activities  for  treatment  teachers  in  the 
first  2 years  (6  out  of  27).  These  same  measures  in  the  control  schools  across  the  3 years  of  the  study 
confirmed  that  use  of  these  activities  in  the  control  schools  constituted  their  standard  practice. 

Of  the  20  child-level  outcomes  representing  the  four  domains  of  Social  and  Emotional  Competence, 
Behavior,  Academics,  and  Perceptions  of  School  Climate  assessed  in  each  of  the  3 years  of  the  study  (a  total 
of  60  results),  none  were  statistically  significant.  A growth  curve  analysis  was  used  to  analyze  the  change  over 
time  in  these  same  outcomes  between  initial  data  collection  and  the  final  outcome  data  collection  at  the  end 
of  the  study.  One  of  the  18  child -level  outcomes  assessed  showed  a significant  beneficial  impact  of  the 
PATHS  program. 

The  SACD  evaluation  did  not  find  evidence  to  support  the  hypothesis  that  PATHS  had  beneficial  impacts  on 
students’  social  and  character  development.  Such  results  could  be  caused  by  the  inability  of  the  program  to 
cause  such  change,  possibly  because  the  theory  of  action  for  the  program  is  incomplete  or  the  activities  to 
carry  out  that  theory  are  not  effective. 

However,  these  results  may  also  be  due  to  the  inability  of  the  evaluation  to  observe  such  a change  due  to  the 
control  condition,  the  level  of  nonparticipation,  or  the  sample  size.  The  control  schools  continued  using  their 
standard  SACD  activities,  and  these  turned  out  to  be  high  in  quantity  and  broad  in  scope.  While  PATHS  had 
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a significant  positive  impact  on  the  amount  and  types  of  SACD  activities,  the  resulting  difference  in  the 
amount  of  SACD  activities  between  the  treatment  and  control  schools  may  not  have  been  large  enough  to 
cause  significant  differences  in  the  student  outcomes.  Second,  nearly  one-half  of  the  students  in  the  sample 
universe  did  not  take  part  because  of  nonconsent  or  noncompletion  of  the  surveys.  As  a determination  could 
not  be  made  as  to  whether  the  students  not  taking  part  significantly  differed  from  those  who  did  take  part, 
the  evaluation’s  results  are  valid  only  for  the  students  who  took  part.  If  the  students  not  taking  part  were 
different,  and  if  they  would  have  responded  better  to  PATHS  than  to  the  SACD  activities  occurring  in  the 
control  schools,  then  the  evaluation  could  have  underestimated  the  program’s  impact.  In  addition,  when 
looking  at  the  percentages  of  the  sample  with  data  across  the  3 study  years  and  the  four  data  sources  (table 
6.4)  there  were  6 comparisons  out  of  16  in  which  the  percentages  of  data  were  greater  for  the  treatment 
reports  than  for  the  control  reports  (all  statistically  significant  differences  at  the  .05  level).  How  these  may 
have  affected  the  results  is  not  clear,  as  the  differences  between  those  with  data  and  those  without  cannot  be 
determined.  Third,  the  sample  size  of  10  schools  and  the  resulting  higher  MDES  compared  to  those  for  the 
multiprogram  evaluation  may  have  reduced  the  likelihood  of  detecting  statistically  significant  effects. 
However,  it  should  be  noted  that  67  percent  of  the  MDES  for  the  60  outcomes  used  in  the  year -by-year 
analysis  were  below  0.25  (42%  were  below  0.20).  In  addition,  only  2 of  the  60  outcomes  were  found  to  be 
substantively  important,  and  both  had  detrimental  impacts. 
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Chapter  7.  The  4Rs  Program  (Reading,  Writing,  Respect,  and 

Resolution) 


New  York  University 
(New  York  City  Site) 


Intervention 

Researchers  at  New  York  University  and  Fordham  University  (New  York  City  site)  evaluated  The  4Rs  Program 
(Reading  Writing  Respect,  and  Resolution j (4Rs)  program  as  implemented  by  staff  developers  from  the 
Morningside  Center  for  Teaching  Social  Responsibility,  the  community-based  nonprofit  organization  that 
developed  the  program.  The  4Rs  program  integrates  the  teaching  and  promotion  of  social  and  emotional 
skills  and  conflict  resolution  lessons  into  the  language  arts  curriculum  and  aims  to  create  a caring  classroom 
community.  Table  7.1  describes  the  4Rs  program’s  general  characteristics  (panel  1),  the  types  of  instruction 
and  strategies  used  (panel  2),  the  professional  development  provided  for  those  implementing  the  program 
(panel  3),  and  the  social  and  character  development  activities  (panel  4)  and  outcomes  (panel  5)  addressed  by 
the  program.  The  program  includes  two  primary  components: 

• The  4Rs  consists  of  a comprehensive,  seven-unit,  21-  to  35-lesson,  literacy-based  curriculum  on 
conflict  resolution  and  social-emotional  learning.  Lesson  units  focus  on  building  community, 
understanding  and  dealing  with  feelings,  respecting  others  and  becoming  a better  listener,  standing 
up  for  oneself  and  learning  to  be  assertive,  understanding  and  dealing  with  conflict,  celebrating 
diversity  and  countering  prejudice,  and  sustaining  a caring  community  and  making  a difference.  Each 
unit  is  based  on  a high-quality  children’s  book,  around  which  discussions,  writing,  and  role-playdng 
activities  are  conducted.  Teachers  engage  students  in  lessons  for  1 hour  at  least  once  per  week  over 
the  school  year.  Teachers  are  encouraged  to  model  and  reinforce  social  skills  throughout  the  day. 

• Teachers  receive  25  hours  of  intensive  introductory  professional  development  designed  to 
(a)  introduce  them  to  the  curricular  units  and  the  children’s  books,  specific  lessons,  and  activities  tied 
to  each  unit;  (b)  give  them  an  opportunity  to  practice  conflict  resolution  skills  at  the  adult  level 
through  role-play  and  experiential  learning;  and  (c)  inspire  them  to  employ  the  ideas  and  skills 
embodied  in  the  curriculum  in  their  own  lives  both  professionally'  and  personally.  This  training  is 
followed  by  ongoing  classroom  coaching  to  support  teachers  in  teaching  the  4Rs  curriculum,  with  a 
minimum  of  12  contacts  in  each  school  year.  Ongoing  coaching  encompasses  class  lesson  modeling 
and  workshops  led  by  a program  staff  developer,  coplanning  and  teaching  of  lessons  by  the  teacher 
and  staff  developer,  and  lesson  observations  and  feedback.  Staff  developers  also  convene  regular 
conferences  with  teachers  in  a one-on-one  format  or  with  a group  of  teachers  from  one  grade. 
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Table  7.1 . The  4Rs  Program  (Reading,  Writing,  Respect,  and  Resolution) 

Panel  1 : General  characteristics 

Target  population 
Universal 

Program  components 
Peer:  In  class 

Parent:  Contact  and  involvement 
Classroom : Lessons 
Schoolwide:  None  or  not  major  focus 
Community:  None  or  not  major  focus 
Training:  Pretraining  and  ongoing 

Level  of  integration 
Core  curriculum 

Flexibility 

Manualized:  Curriculum  guidebook 

Adaptability:  Less  adaptable 

See  notes  at  end  of  table. 


Panel  2:  Description  of  instruction  and  strategies 

Classroom 

Lessons 

Who  delivers:  T eacher 

Activities  and  tools:  Story  reading,  role-playing,  discussions,  writing,  reflections,  sharing  exercises, 
brainstorming,  songs,  worksheets 

Content:  Literacy  focus;  building  community,  feelings,  listening,  assertiveness,  problem  solving,  diversity, 
making  a difference 

Frequency:  1-hour  lessons  at  least  once  per  week;  minimum  of  21  lessons 
Strategies 

Who  delivers:  T eacher 

Activities  and  tools:  Modeling  and  reinforcement  of  social  skills 
Frequency:  Daily 

Supplement  to  classroom 

Family  Connections  homework  and  parent  workshops 
Schoolwide  activities 

None 

See  notes  at  end  of  table. 
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Table  7.1.  The  4Rs  Program  (Reading,  Writing,  Respect,  and  Resolution) — Continued 

Panel  3:  Professional  development 

Pre-implementation 

Teachers 

Content:  Course  to  learn  curriculum,  improve  on  own  skills,  create  a vision  of  community,  set  consistent 
Rules,  and  behavior  management 
Duration:  25  hours 
Other 

Content:  None 
Duration:  None 

Ongoing  consultation 
Teachers 

Content:  Training  and  support  in  mediation;  negotiation;  role-playing;  class  meetings;  behavior 
management 

Duration:  Individualized  coaching  a minimum  of  12  times  per  year;  3-day  summer  institute 
Other 

Content:  None 

Duration:  None 

See  notes  at  end  of  table. 


Panel  4:  Activities  for  SACD  goals 


Violence  prevention  and  peace  promotion 

✓ 

Risk  prevention  and  health  promotion 

✓ 

Social  and  emotional  development 

✓ 

Civic  responsibility  and  community  service 

Character  education 

V 

Behavior  management 

✓ 

Tolerance  and  diversity 

V 

See  notes  at  end  of  table. 

Panel  5:  SACD  outcomes  addressed 

Engagement  with  Learning 

s 

Empathy 

✓ 

Academic  Competence  and  Motivation 

Positive  School  Orientation 

Altruistic  Behavior 

Negative  School  Orientation 

Positive  Social  Behavior 

✓ 

Student  Afraid  at  School 

Problem  Behavior 

Victimization  at  School 

✓ 

Self-Efficacy  for  Peer  Interactions 

Feelings  of  Safety 

✓ 

Normative  Beliefs  About  Aggression 

Student  Support  for  Teachers 

NOTE:  Abbreviations  are 
s-.  Activity  or  outcome  addressed 
Blank  cell:  Activity  or  outcome  not  addressed 
SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Sample  and  Random  Assignment 

The  New  York  City  research  team  recruited  a total  of  14  public  elementary  schools  in  eight  school  districts  in 
four  boroughs  in  New  York  City.51  These  14  schools  were  randomly  assigned  to  treatment  and  control 
conditions  prior  to  the  fall  2004  data  collection.  A two-step  process  was  used.  First,  a computer-generated 
pairwise  matching  algorithm  developed  by  Mathematica  Policy  Research,  Inc.  (MPR)  was  used  to  identify  the 
best  pairwise  matches  across  the  14  schools  based  on  variables  identified  by  the  New  York  University 
research  team.  The  variables  used  in  the  pairwise  matching  for  the  New  York  City  site  included  the  following: 
(a)  average  spending  per  student,  (b)  percentage  of  part-time  special  education  students,  (c)  percentage  of 
students  at  level  on  the  English  language  arts  standardized  achievement  test,  (d)  attendance,  (e)  percentage  of 
students  eligible  for  free  school  lunch,  (f)  percentage  of  students  who  were  recent  immigrants,  (g)  number  of 
students,  (h)  percentage  of  Hispanic  students,  (i)  percentage  of  Asian/ Other  students,  (j)  percentage  of  male 
students,  (k)  student  stability,  (1)  average  days  teacher  absent,  (m)  percentage  of  fully  licensed  teachers, 
(n)  percentage  of  teachers  with  more  than  2 years  of  teaching  experience,  (o)  direct  total  classroom 
instruction  expenditure,  (p)  percentage  of  foreign-born  students,  (q)  Bloomberg  School  list,  (r)  No  Child  Left 
Behind  improvement  status  school,  (s)  teacher  organizational  readiness,  and  (t)  school  organizational 
readiness.  In  early  spring  2004,  as  part  of  the  initial  school  recruitment  process,  organizational  readiness  was 
measured  by  assessing  such  important  school  dimensions  as  principal  leadership  style,  openness  of 
communication,  administrative/ teacher  buy-in,  administrative  and  staff  stability,  number  and  degree  of  other 
programs,  demands  on  teacher  time,  and  amount  of  professional  development.  Second,  using  a computer- 
based  random  number  generator,  1 school  in  each  matched  pair  was  assigned  to  either  the  intervention  or 
control  condition.  Seven  schools  received  the  4Rs  program  and  7 schools  acted  as  control  schools  and 
continued  to  implement  the  social  and  character  development  activities  that  constituted  their  standard 
practice.  Assignment  to  treatment  or  control  condition  was  at  the  school  level  and  therefore  limited  the  risk 
of  contamination  between  treatment  and  control  classrooms. 

The  original  student  sample  (the  cohort  of  students  in  the  third  grade  in  the  14  schools  in  fall  2004) 
numbered  1,202  students  (652  treatment  and  550  control).  Table  7.2  documents  the  change  in  the  sample 
over  the  three  spring  follow-up  data  collection  periods.  Over  time,  the  percentage  of  new  entrants  to  the 
cohort  increased,  eventually  making  up  33  percent  of  the  sample  by  the  spring  of  Year  3.  There  were  no 
statistically  significant  differences  between  the  treatment  and  control  groups  in  the  number  of  new  entrants. 
The  percentage  of  the  sample  made  up  of  the  original  cohort  further  declined  as  students  left  the  schools.  By 
Year  3,  approximately  41  percent  of  the  original  sample  had  left. 


51  The  New  York  University  research  team  recruited  18  schools  but  only  14  were  included  in  the  multiprogram 
evaluation. 
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Year  1 Year  2 Year  3 Year  4 

(Fall  3rd  grade) (Spring  3rd  grade) (Spring  4th  grade)  (Spring  5th  grade) 


Treat-  Treat-  Treat-  Treat- 


Characteristic 

Total 

ment 

Control 

Total 

ment 

Control 

Total 

ment 

Control 

Total 

ment 

Control 

School  sample  size 

14 

7 

7 

14 

7 

7 

14 

7 

7 

14 

7 

7 

Student  sample  size 

1,202 

652 

550 

1,194 

647 

547 

1,109 

599 

510 

1,065 

556 

509 

Stayers 

t 

T 

t 

1,116 

603 

513 

882 

469 

413 

710 

373 

337 

New  entrants 

t 

T 

t 

78 

44 

34 

227 

130 

97 

355 

183 

172 

New  entrants  as  a percent 
of  spring  enrollment 

t 

T 

t 

6.5 

6.8 

6.2 

20.5 

21.7 

19.0 

33.3 

32.9 

33.8 

Total  leavers  (from  original 
cohort) 

t 

t 

t 

86 

49 

37 

320 

183 

137 

492 

279 

213 

Leavers  as  percent  of  fall 
2004  enrollment 

t 

t 

t 

7.2 

7.5 

6.7 

26.6 

28.1 

24.9 

40.9 

42.8 

38.7 

Number  of  students  per 
school  (mean) 

86 

93 

79 

85 

92 

78 

80 

86 

73 

76 

79 

73 

Range  of  number  of  students 
per  school 

58-149 

58-149 

64-97 

58-147 

58-147 

66-94 

46-136 

52-136 

46-86 

43-126 

49-126 

43-101 

t Not  applicable. 

NOTE:  No  statistically  significant  differences  were  found  between  values  for  treatment  and  control  groups. 
SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Implementation 

Training 

The  intervention  teachers  received  25  hours  of  program  implementation  training  prior  to  the  beginning  of  the 
school  year  (table  7.1,  panel  3).  On  average,  teachers  received  an  additional  2.5  days  of  training  during  the 
school  year.  Onsite  coaching  occurred  throughout  the  year,  with  the  average  school  receiving  about  1 day  per 
week  of  staff  development  visits.  Coaching  visits  typically  included  a classroom  experience  (a  demonstration 
lesson  by  the  staff  developer,  a lesson  cofacilitated  by  the  staff  developer  and  the  teacher,  or  a lesson  taught 
by  the  teacher  and  observed  by  the  staff  developer)  and  a conference  with  the  teacher  for  debriefing. 

Data  Collection 

MPR  collected  the  child,  teacher,  and  school  data  for  the  New  York  City  site.  Table  7.3  shows  the  school  year 
milestones  and  dates  of  implementation  for  the  New  York  City  site.  Data  were  collected  in  the  fall  and  spring 
of  the  first  2 years  and  the  spring  of  Year  3.  The  fall  2004  data  collection  began  on  October  18,  2004,  and 
ended  on  November  19,  2004.  The  average  time  frame  from  the  beginning  of  program  implementation  to  the 
beginning  of  fall  data  collection  was  4 weeks.  As  a result,  initial  data  collection  took  place  after  4Rs 
implementation  began.  Therefore,  these  data  provide  a measure  of  the  students,  teachers,  and  schools  near 
the  beginning  of  the  school  year,  at  a time  when  the  program  had  been  operating  for  a relatively  short  period 
of  time.  The  spring  data  collection  window  was  from  March  21,  2005,  to  April  15,  2005.  The  4Rs  program 
had  been  implemented  for  26  weeks  at  the  time  of  the  spring  data  collection  and  for  17  weeks  from  the  end 
of  the  fall  data  collection.  Year  2 followed  a similar  pattern,  with  implementation  occurring  at  the  start  of  the 
school  year,  fall  data  collection  occurring  5 weeks  later,  and  spring  data  collection  occurring  20  weeks  after 
fall  data  collection  (and  29  weeks  after  the  start  of  implementation).  In  spring  2007,  data  collection  occurred 
29  weeks  after  the  start  of  implementation.  Data  collection  took  from  3 to  5 weeks  at  each  collection  point. 
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Table  7.3.  Data  collection  dates — 4Rs 


Year  1 Year  1 Year  2 Year  2 Year  3 

Data  collection  schedule  (Fall  3rd  grade)  (Spring  3rd  grade)  (Fall  4th  grade)  (Spring  4th  grade)  (Spring  5th  grade) 


School  sample  size 

14 

14 

14 

14 

14 

School  year  dates 
First  day  of  school 
Start  of  implementation 
Last  day  of  school 

9/7/04 

9/20/04 

t 

t 

t 

6/28/05 

9/8/05 
First  day 
t 

t 

t 

6/27/06 

9/5/06 
First  day 
6/27/07 

Data  collection 
Start 
End 

10/18/04 

11/19/04 

3/21/05 
4/1 5/05 

10/11/05 

11/16/05 

3/27/06 

5/8/06 

3/26/07 

4/26/07 

Calendar  weeks  from 
program  implementation 
to  start  of  fall  2004 
data  collection 

4 

t 

t 

t 

t 

Calendar  weeks  from 
start  of  school  to  start  of 
fall  2004  data  collection 

6 

t 

5 

t 

t 

Calendar  weeks  from  end 
of  fall  data  collection  to 
start  of  Spring  data 
collection 

t 

17 

t 

20 

t 

Calendar  weeks  from 
program  implementation 
to  start  of  spring  data 
collection 

t 

26 

t 

29 

29 

t Not  applicable. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Consent  Rates,  Completion  Rates,  and  Percentage  of  Sample  With  Data 

The  actual  number  of  student,  primary  caregiver,  and  teacher  reports  available  for  analysis  was  smaller  than 
the  number  in  the  sample  because  consent  and  completion  rates  were  below  100  percent.  Primary  caregivers 
had  to  provide  consent  before  children  could  complete  the  Child  Report,  before  their  child’s  teacher  could 
complete  the  Teacher  Report  on  Student,  and  before  they  themselves  completed  the  Primary  Caregiver 
Report.  Teachers  also  had  to  provide  consent  before  completing  the  Teacher  Report  on  Classroom  and 
School.  There  were  no  statistically  significant  differences  between  treatment  and  control  groups  in  consent 
rates  for  any  of  these  reports. 

Of  those  with  consent,  not  all  completed  their  respective  reports.  Table  7.4  shows  the  consent  rates, 
completion  rates,  and  percentages  of  sample  with  data  for  each  of  the  four  reports  over  the  3 years.  For  the 
Child  Report  and  two  teacher  reports,  completion  rates  ranged  from  91  percent  to  100  percent,  with  two 
statistically  significant  differences  between  treatment  and  control  groups  in  completion  rates.  In  Year  3,  more 
Child  Reports  and,  in  Year  2,  more  Teacher  Reports  on  Students  were  completed  in  the  treatment  group 
compared  to  the  control  group.  For  the  Primary  Caregiver  Report,  the  completion  rates  dropped  over  time 
from  85  percent  to  58  percent.  There  were  no  statistically  significant  differences  for  the  Primary  Caregiver 
Report  completion  rates. 

The  percentages  of  the  sample  with  Child  Report  data  ranged  from  54  percent  to  64  percent  over  the  3 years, 
with  a statistically  significant  difference  in  Year  1,  when  more  students  in  the  treatment  group  had  data 
compared  to  the  control  group.  The  percentages  of  students  with  information  from  the  Teacher  Report  on 
Student  ranged  from  52  percent  to  67  percent,  with  a statistically  significant  difference  between  treatment  and 
control  conditions  in  Year  1 favoring  the  treatment  group.  The  percentages  of  students  with  data  from  the 
Primary  Caregiver  Report  ranged  from  32  percent  to  42  percent.  In  general,  this  percentage  declined  over 
time,  although  the  highest  rates  were  seen  in  the  spring  of  2005.  There  were  no  statistically  significant 
differences  between  treatment  and  control  groups  for  this  data.  The  percentages  of  teachers  with  data  from 
the  Teacher  Report  on  Classroom  and  School  ranged  from  89  percent  to  98  percent,  with  no  statistically 
significant  difference  between  treatment  and  control  teachers. 
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Table  7.4.  Consent  rates,  completion  rates,  and  percentage  of  sample  with  data  from  each  report— 4Rs 


4^ 

o 


Year  1 Year  1 Year  2 Year  3 

(Fall  3rd  grade)  (Spring  3rd  grade)  (Spring  4th  grade)  (Spring  5th  grade) 

Treat-  Treat-  Treat-  Treat- 


Report 

Total 

ment 

Control 

Total 

ment 

Control 

Total 

ment 

Control 

Total 

ment 

Control 

Student  sample  size 

1202 

652 

550 

1194 

647 

547 

1109 

599 

510 

1065 

556 

509 

Child  Report  (percent) 

Primary  caregiver  consent  rate 

56.8 

58.4 

54.9 

61.5 

64.1 

58.3 

66.3 

67.3 

65.1 

64.3 

64.6 

64.0 

Student  completion  rate 

96.6 

95.8 

97.7 

96.2 

97.1 

95.0 

92.4 

93.3 

91.3 

97.1 

98.3* 

95.7 

Students  with  data1 

54.9 

56.0 

53.6 

59.1 

62.3* 

55.4 

61.2 

62.8 

59.4 

62.4 

63.5 

61.3 

Primary  Caregiver  Report  (percent) 

Primary  caregiver  consent  rate 

47.8 

49.4 

46.0 

52.8 

54.9 

50.5 

58.3 

58.8 

57.8 

57.1 

56.3 

58.0 

Primary  caregiver  completion  rate 

85.7 

85.4 

86.2 

78.9 

76.3 

82.2 

70.3 

69.6 

71.2 

58.1 

56.9 

59.3 

Primary  caregivers  with  data1 

41.0 

42.2 

39.6 

41.7 

41.9 

41.5 

41.0 

40.9 

41.2 

33.1 

32.0 

34.4 

Teacher  Report  on  Student  (percent) 

Primary  caregiver  consent  rate2 

56.8 

58.4 

54.9 

61.5 

64.1 

58.3 

66.3 

67.3 

65.1 

64.3 

64.6 

64.0 

Teacher  completion  rate 

94.7 

95.0 

94.4 

97.0 

97.3 

96.6 

98.8 

100.0*** 

97.3 

99.7 

99.7 

99.7 

Students  with  data1 

53.8 

55.5 

51.8 

59.6 

62.4* 

56.3 

65.5 

67.3 

63.3 

64.1 

64.4 

63.9 

Teacher  Report  on  Classroom  and  School 
(3rd-  to  5th-grade  teachers)  (percent) 

Teacher  consent  rate 

96.3 

98.0 

94.5 

94.8 

95.0 

94.6 

97.2 

96.8 

97.6 

97.0 

98.9 

94.9 

Teacher  completion  rate 

96.2 

93.9 

98.8 

95.6 

93.7 

97.7 

100.0 

100.0 

100.0 

99.4 

98.9 

100.0 

Teachers  with  data1 

92.7 

92.0 

93.4 

90.6 

89.0 

92.4 

97.2 

96.8 

97.6 

96.4 

97.8 

94.9 

* Treatment  group  significantly  different  from  control  group  at  the  .05  level. 

***  Treatment  group  significantly  different  from  control  group  at  the  .001  level. 

1 Calculated  as  consent  rate  x completion  rate. 

2The  primary  caregiver  consent  rates  for  the  Child  Report  and  the  Teacher  Report  on  Student  are  identical,  as  the  primary  caregiver  gave  consent  to  both  together. 
SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Responses  from  students  in  the  original  cohort  (stayers)  and  new  entrants  in  the  4Rs  sample  were  examined 
to  investigate  possible  differences  between  the  two  groups  in  consent  rates,  completion  rates,  and  the 
percentages  of  sample  with  data  that  might  affect  outcome  data  (table  7.5).  In  Years  2 and  3,  consent  rates  for 
the  child  and  the  primary  caregiver  were  significantly  higher  for  the  stayers  than  for  the  new  entrants  (by  17 
to  23  percentage  points),  which  resulted  in  significandy  larger  percentages  of  data  on  both  the  Child  Report 
(by  15  to  21  percentage  points)  and  the  Teacher  Report  on  Student  (by  17  to  23  percentage  points).  In 
contrast,  more  primary  caregivers  in  the  new  entrants  group  compared  to  the  stayers  completed  the  Primary 
Caregiver  report  (by  9 to  17  percentage  points)  in  all  3 years. 
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Table  7.5.  Consent  rates,  completion  rates,  and  percentage  of  sample  with  data:  Stayers  versus  new  entrants — 4Rs 


Year  1 Year  2 Year  3 

(Spring  3rd  grade)  (Spring  4th  grade) (Spring  5th  grade) 

New  New  New 


Report 

Total 

Stayers 

entrants 

Total 

Stayers 

entrants 

Total 

Stayers 

entrants 

Student  sample  size 

1,194 

1,116 

78 

1,109 

882 

227 

1,065 

710 

355 

Child  Report  (percent) 

Primary  caregiver  consent  rate 

61.5 

61.8 

56.4 

66.3 

69.7*** 

52.9 

64.3 

72  1 *** 

48.7 

Student  completion  rate 

96.2 

96.1 

97.7 

92.4 

92.2 

93.3 

97.1 

96.5 

98.8 

Students  with  data1 

59.1 

59.4 

55.1 

61.2 

64.3*** 

49.3 

62.4 

69.6*** 

48.2 

Primary  Caregiver  Report  (percent) 

Primary  caregiver  consent  rate 

52.8 

53.0 

50.0 

58.3 

60.3** 

50.7 

57.1 

61.8*** 

47.6 

Primary  caregiver  completion  rate 

78.9 

77.9* 

94.9 

70.3 

67.9** 

81.7 

59.4 

56.9* 

65.7 

Primary  caregivers  with  data1 

41.7 

41.3 

47.4 

41.0 

40.9 

41.4 

33.9 

35.2 

31.3 

Teacher  Report  on  Student  (percent) 

Primary  caregiver  consent  rate2 

61.5 

61.8 

56.4 

66.3 

69.7*** 

52.9 

64.3 

72  1 *** 

48.7 

Teacher  completion  rate 

97.0 

97.0 

97.7 

98.8 

98.9 

98.3 

99.7 

99.6 

100.0 

Students  with  data1 

59.6 

59.9 

55.1 

65.5 

68.9*** 

52.0 

64.1 

71.8*** 

48.7 

* Stayers  significantly  different  from  new  entrants  at  the  .05  level. 

**  Stayers  significantly  different  from  new  entrants  at  the  .01  level. 

***  Stayers  significantly  different  from  new  entrants  at  the  .001  level. 

1 Calculated  as  consent  rate  x completion  rate. 

2The  primary  caregiver  consent  rates  for  the  Child  Report  and  the  Teacher  Report  on  Student  are  identical,  as  the  primary  caregiver  gave  consent  to  both  together. 
SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Fidelity  of  Implementation 

Each  year,  the  seven  4Rs  treatment  schools  were  independently  rated  for  quantity  and  quality  of  program 
implementation  by  two  raters  from  the  research  team.  The  global  measure  of  fidelity  for  the  multisite  study 
was  used;  inter-rater  reliability  was  measured  using  Cronbach’s  alpha  (0.84  in  Year  1,  0.78  in  Year  2,  and  0.72 
in  Year  3).  The  ratings  were  combined  into  a single  consensus  rating  and  used  to  identify  schools  with  high 
implementation  fidelity.  In  Years  1 and  2,  two  treatment  schools  and,  in  Year  3,  three  treatment  schools,  were 
identified  as  having  high  fidelity.  Cohen’s  kappa  was  used  as  the  measure  of  agreement  when  identifying 
schools  as  high  fidelity  and  it  equaled  0.70  in  Year  1,  0.59  in  Year  2,  and  0.30  in  Year  3. 

Initial  Characteristics 

This  section  examines  the  initial  characteristics  of  the  students,  teachers,  and  schools  participating  in  the  4Rs 
evaluation.  These  characteristics  were  collected  from  students  who  were  enrolled  in  the  third  grade  at  the 
study  schools  in  fall  2004,  as  well  as  from  their  primary  caregivers  and  third-grade  teachers.  In  addition, 
third-,  fourth-,  and  fifth-grade  teachers  and  principals  in  the  study  schools  provided  information  about 
activities  related  to  social  and  character  development  in  these  schools.  Documenting  the  characteristics  of 
students,  teachers,  and  schools  and  initial  measures  of  key  outcomes  at  a point  before  the  intervention  had 
been  operating  for  an  extended  period  helped  to  determine  whether  the  random  assignment  of  schools  to 
treatment  and  control  status  produced  treatment  and  control  groups  with  similar  distributions  of  observed 
characteristics.  As  noted  in  the  following  discussion,  there  were  8 significant  differences  in  the  observed 
characteristics,  7 of  which  (out  of  62  comparisons,  with  3 expected  to  be  significant  by  chance)  reflected 
differences  between  the  treatment  and  control  students,  teachers,  and  schools  in  the  school  and  classroom 
use  of  SACD  activities. 

Characteristics  of  Children,  Their  Families,  and  Communities 

There  were  no  significant  differences  between  the  treatment  and  control  groups  in  the  observed  student, 
caregiver,  and  community  characteristics  (table  7.6).  For  students,  the  mean  age  was  8.1  years.  The  sample 
contained  roughly  equal  percentages  of  girls  (52%)  and  boys  (48%).  The  sample  was  ethnically  diverse,  with 
White  non-Hispanic  students  making  up  5 percent  of  the  sample,  Black  non-Hispanic  students  making  up  41 
percent  of  the  sample,  and  Hispanic  students  making  up  46  percent  of  the  sample. 

The  sample  was  also  diverse  in  its  levels  of  family  income,  education  levels  of  primary  caregivers  of  the 
children  in  the  sample,  and  family  situation.  For  the  total  sample,  58  percent  of  children  lived  in  a household 
where  the  income  was  135  percent  of  the  federal  poverty  level  or  lower,  which  is  the  income  threshold  for 
eligibility  for  free  school  meals.  Twenty-eight  percent  of  primary  caregivers  had  not  completed  high  school. 
Less  than  half  of  the  children  (48%)  lived  with  both  their  mother  and  their  father.  There  were  no  significant 
differences  in  these  characteristics  between  the  treatment  and  control  groups. 

The  mean  values  of  the  outcomes  for  children’s  behavior  and  attitudes  as  reported  by  the  primary  caregiver, 
child,  and  teacher  at  initial  data  collection  in  fall  2004  are  shown  in  table  7.7.  There  were  no  significant 
differences  between  the  treatment  and  control  groups. 
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Table  7.6.  Initial  characteristics  of  children,  their  families,  and  communities — 4Rs 


Characteristic 

Total 

Treatment 

Control 

Student  sample  size 

493 

275 

218 

Student  demographics 

Gender  (percent) 

Male 

48.3 

49.3 

47.3 

Female 

51.7 

50.7 

52.7 

Race/ethnicity  (percent) 

White  (non-Hispanic) 

5.3 

7.0 

3.6 

Black  (non-Hispanic) 

40.5 

43.4 

37.6 

Hispanic 

45.9 

43.1 

48.7 

Other 

8.3 

6.5 

10.1 

Age  (in  years)  (mean) 

8.1 

8.1 

8.2 

Primary  caregiver  and  family  characteristics 

Primary  caregiver’s  age  (in  years)  (mean) 

35.1 

36.0 

34.3 

Primary  caregiver’s  race/ethnicity  (percent) 

White  (non-Hispanic) 

7.1 

7.9 

6.2 

Black  (non-Hispanic) 

41.1 

44.4 

37.8 

Hispanic 

43.4 

41.0 

45.8 

Other 

8.4 

6.7 

10.2 

Primary  caregiver’s  education  (percent) 

Did  not  complete  high  school 

28.0 

27.2 

28.8 

Completed  high  school  or  equivalent 

27.5 

25.9 

29.1 

Some  college 

32.2 

31.2 

33.3 

Bachelor’s  or  higher  degree 

12.3 

15.8 

8.8 

Primary  caregiver’s  employment  (percent) 

Full-time 

38.9 

37.5 

40.3 

Other 

61.1 

62.5 

59.7 

Primary  caregiver’s  marital  status  (percent) 

Married 

40.3 

41.6 

39.0 

Other 

59.7 

58.4 

61.0 

Students  who  live  in  one  household  (percent) 

91.7 

92.1 

91.4 

Number  of  individuals  in  household  (mean) 

4.7 

4.6 

4.9 

Primary  caregiver’s  relationship  to  child  (percent) 

Mother  (stepmother) 

86.5 

85.9 

87.2 

Father  (stepfather) 

8.6 

7.3 

9.8 

Other  relative/nonrelative 

4.9 

6.8 

3.0 

See  notes  at  end  of  table. 
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Table  7.6.  Initial  characteristics  of  children,  their  families,  and  communities — 4Rs — Continued 


Characteristic 

Total 

Treatment 

Control 

Student  lives  with  (percent) 

Mother  (stepmother)  and  father  (stepfather) 

47.9 

46.7 

49.0 

Mother  (stepmother)  only;  father  (stepfather)  not  present 

46.1 

47.1 

45.0 

Father  (stepfather)  only;  mother  (stepmother)  not  present 

2.9 

1.9 

3.9 

Other  relative/nonrelative,  parents  not  present 

3.2 

4.3 

2.0 

Highest  education  of  anyone  in  household  (percent) 

Did  not  complete  high  school 

21.0 

18.4 

23.6 

Completed  high  school  or  equivalent 

28.9 

26.9 

31.0 

Some  college 

32.6 

32.3 

32.9 

Bachelor’s  or  higher  degree 

17.4 

22.4 

12.5 

Total  household  income  (percent) 

Less  than  $20,000 

51.7 

51.0 

52.5 

$20,000  to  $39,999 

25.9 

24.2 

27.7 

$40,000  to  $59,999 

9.9 

8.5 

11.3 

$60,000  or  more 

12.5 

16.4 

8.6 

Income-to-poverty-threshold  ratio — Below  135  percent  (percent) 

57.8 

55.2 

60.5 

Income-to-poverty-threshold  ratio — 135  to  185  percent  (percent) 

19.3 

20.1 

18.5 

Income-to-poverty-threshold  ratio — Above  185  percent  (percent) 

22.9 

24.8 

21.1 

Alabama  Parenting  Questionnaire-Poor  Monitoring  and 

Supervision  Subscale  (mean) 

1.2 

1.2 

1.2 

Alabama  Parenting  Questionnaire-Positive  Parenting 

Subscale  (mean) 

3.5 

3.5 

3.5 

Confusion,  Hubbub,  and  Order  Scale  (mean) 

2.2 

2.1 

2.2 

Community  characteristics  (mean) 

Community  Risks  Scale 

2.0 

1.9 

2.0 

Community  Resources  Scale 

2.8 

2.8 

2.8 

Child-Centered  Social  Control  Scale 

2.7 

2.7 

2.8 

NOTE:  No  statistically  significant  differences  were  found  between  values  for  treatment  and  control  groups.  Weights,  which  assign 
equal  weight  to  each  school  within  the  program,  were  used  in  producing  the  treatment,  control,  and  overall  means.  Statistical  tests 
were  conducted  using  regressions  that  included  program  indicators  to  account  for  the  sample  design  and  adjusted  for  clustering  at 
the  school  level. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Table  7.7.  Mean  scores  and  standard  deviations  for  initial  outcome  measures  of  sample — 4Rs 


Outcome  measure-Report 

Range 

Total 

Mean 

SD 

Treatment 
Mean  SD 

Control 

Mean 

SD 

Social  and  Emotional  Competence  Domain 

Self-Efficacy  for  Peer  Interaction-CR 

1-4 

2.9 

0.7 

2.9 

0.6 

2.9 

0.7 

Normative  Beliefs  About  Aggression-CR 

1-4 

1.3 

0.5 

1.3 

0.5 

1.3 

0.6 

Empathy-CR 

1-3 

2.4 

0.4 

2.4 

0.4 

2.4 

0.4 

Behavior  Domain 

Altruistic  Behavior-CR 

0-3 

1.6 

0.8 

1.6 

0.8 

1.6 

0.8 

Altruistic  Behavior-TRS 

1-4 

1.5 

0.6 

1.5 

0.6 

1.5 

0.6 

Altruistic  Behavior-PCR 

1-4 

2.3 

0.8 

2.4 

0.8 

2.3 

0.8 

Positive  Social  Behavior-TRS 

1-4 

2.8 

0.7 

2.8 

0.7 

2.8 

0.7 

Positive  Social  Behavior-PCR 

1-4 

2.9 

0.5 

3.0 

0.5 

2.9 

0.5 

Problem  Behavior-CR 

0-3 

0.4 

0.5 

0.3 

0.5 

0.4 

0.6 

Problem  Behavior-TRS 

1-4 

1.5 

0.5 

1.5 

0.6 

1.4 

0.4 

Problem  Behavior-PCR 

1-4 

1.5 

0.3 

1.5 

0.3 

1.5 

0.3 

ADHD-Related  Behavior-TRS 

1-4 

1.8 

0.7 

1.8 

0.7 

1.8 

0.6 

Academics  Domain 

Academic  Competence  and  Motivation-TRS 

1-5 

2.6 

0.8 

2.6 

0.9 

2.6 

0.8 

Engagement  with  Learning-CR 

1-4 

3.6 

0.7 

3.6 

0.7 

3.7 

0.6 

Perceptions  of  School  Climate  Domain 

Positive  School  Orientation-CR 

1-4 

3.0 

0.7 

3.0 

0.8 

3.1 

0.7 

Negative  School  Orientation-CR 

1-4 

2.1 

0.6 

2.1 

0.6 

2.1 

0.7 

Student  Afraid  at  School-CR 

1-4 

2.6 

0.9 

2.6 

1.0 

2.6 

0.9 

Victimization  at  School-CR 

0-3 

0.9 

0.8 

0.8 

0.8 

0.9 

0.8 

Student  sample  size — PCR 

497 

228 

269 

Student  sample  size — CR 

521 

238 

283 

Student  sample  size — TRS 

525 

248 

277 

NOTE:  Abbreviations  are 
CR:  Child  Report 
PCR:  Primary  Caregiver  Report 
TRS:  Teacher  Report  on  Student 
ADHD:  Attention  deficit  hyperactivity  disorder 
SD:  Standard  deviation 

No  statistically  significant  differences  were  found  between  values  for  treatment  and  control  groups.  Weights,  which  assign  equal 
weight  to  each  school  within  the  program,  were  used  in  producing  the  treatment,  control,  and  overall  means.  Statistical  tests  were 
conducted  using  regressions  that  included  program  indicators  to  account  for  the  sample  design  and  adjusted  for  clustering  at  the 
school  level.  Sample  size  may  differ  for  some  outcomes  due  to  nonresponse. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Characteristics  of  Teachers  and  Schools 

Table  7.8  describes  the  third-,  fourth-,  and  fifth-grade  teachers  at  the  study  schools.  Half  the  members  of  this 
sample  were  White  non-Hispanic  (50%),  and  most  were  female  (86%).  They  had  an  average  of  8.5  years  of 
total  teaching  experience,  and  there  was  a statistically  significant  difference  between  the  treatment  and  control 
groups,  with  control  teachers  having  more  teaching  experience  than  treatment  teachers  (10  years  versus  7 
years).  The  majority  of  the  teachers  (78%)  held  an  advanced  or  specialist  degree. 

Data  related  to  school  characteristics  were  drawn  from  the  Common  Core  of  Data  in  order  to  compare 
treatment  and  control  schools.  There  were  no  significant  differences  between  the  two  groups  of  schools  in 
terms  of  student  composition  (race/ ethnicity  and  school  lunch  eligibility),  number  of  students  enrolled, 
number  of  full-time  teachers,  Title  I status,  or  number  of  years  the  principal  had  been  at  the  school  (see  table 
7.9).  In  addition,  there  were  no  significant  differences  between  treatment  and  control  schools  in  terms  of 
location  (urban,  suburban,  or  rural)  or  lowest  and  highest  grade  offered  (these  data  are  not  shown  in  a table). 

Table  7.8.  Initial  characteristics  of  teachers  in  sample — 4Rs 


Characteristic 

Total 

Treatment 

Control 

Teacher  sample  size 

177 

92 

85 

Gender  (percent) 

Male 

14.4 

14.8 

13.9 

Female 

85.6 

85.2 

86.1 

Race/ethnicity  (percent) 

White  (non-Hispanic) 

50.0 

47.9 

52.2 

Other 

50.0 

52.1 

47.8 

Number  of  years  teaching  experience  (mean) 

8.5 

7.0* 

10.0 

Number  of  years  teaching  experience  in  this  school  (mean) 

5.6 

5.0 

6.2 

Type  of  teaching  certificate  (percent) 

Regular  state  certificate  or  advanced  professional  certificate 

68.0 

62.4 

73.5 

Other 

32.0 

37.6 

26.5 

Education  (percent) 

Bachelor’s  degree 

22.0 

18.7 

25.2 

Advanced  degree/other 

78.0 

81.3 

74.8 

* Treatment  group  significantly  different  from  control  group  at  the  .05  level. 

NOTE:  Weights,  which  assign  equal  weight  to  each  school  within  the  program,  were  used  in  producing  the  treatment,  control,  and 
overall  means.  Statistical  tests  were  conducted  using  regressions  that  included  program  indicators  to  account  for  the  sample  design 
and  adjusted  for  clustering  at  the  school  level.  Sample  size  may  differ  for  some  outcomes  due  to  nonresponse. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Table  7.9.  Initial  characteristics  of  schools  in  sample — 4Rs 


Characteristic 

Total 

Treatment 

Control 

School  sample  size 

14 

7 

7 

Student  race/ethnicity  (percent) 

White  (non-Hispanic) 

6.7 

8.8 

4.6 

Black  (non-Hispanic) 

47.1 

47.7 

46.5 

Hispanic 

40.1 

37.2 

42.9 

Other 

6.1 

6.3 

6.0 

Students  eligible  for  free  or  reduced-price  lunch  (percent) 

85.1 

85.9 

84.5 

Number  of  students  enrolled  (mean) 

604.2 

608.6 

599.9 

Number  of  full-time  teachers  (mean) 

44.2 

43.4 

44.8 

Title  1 status  (percent) 

Title  1 eligible  school 

92.9 

85.7 

100.0 

Schoolwide  Title  1 

76.9 

83.3 

71.4 

Number  of  years  principal  has  been  at  this  school  (mean) 

6.4 

7.4 

5.0 

NOTE:  No  statistically  significant  differences  were  found  between  values  for  treatment  and  control  groups.  Weights,  which  assign 
equal  weight  to  each  school  within  the  program,  were  used  in  producing  the  treatment,  control,  and  overall  means.  Statistical  tests 
were  conducted  using  regressions  that  included  program  indicators  to  account  for  the  sample  design  and  adjusted  for  clustering  at 
the  school  level.  Sample  size  may  differ  for  some  outcomes  due  to  nonresponse. 

SOURCE:  NCES  Common  Core  of  Data  (2003-04),  the  Social  and  Character  Development  (SACD)  Research  Program. 


In  the  Teacher  Report  on  Classroom  and  School,  teachers  reported  on  nine  dimensions  of  school 
environment  (these  data  are  not  shown  in  a table):  feelings  of  safety,  adequacy  of  resources,  student  support, 
freedom  to  teach  as  desired,  affiliation  with  and  ties  to  colleagues,  innovation  regarding  new  approaches  to 
teaching,  professional  interest,  participatory  decisionmaking,  and  work  pressure.  There  were  no  statistically 
significant  differences  between  treatment  and  control  schools. 

The  Level  of  SACD  in  the  Schools  Near  the  Beginning  of  the  Study 

During  the  initial  data  collection,  principals  and  teachers  reported  on  the  SACD  activities  used  in  the  schools 
and  classrooms,  the  availability  of  SACD  materials,  and  the  professional  development  provided  on  SACD. 
Table  7.10  shows  that  the  majority  of  the  school  principals  reported  activities  to  promote  six  social  and 
character  development  goals:  violence  prevention  and  peace  promotion  (93%),  social  and  emotional 
development  (100%),  character  education  (100%),  tolerance  and  diversity  (100%),  risk  prevention  and  health 
promotion  (93%),  and  civic  responsibility  and  community  service  (100%).  In  addition,  64  percent  of  the 
principals  reported  activities  directed  toward  behavior  management.  There  were  no  statistically  significant 
differences  between  the  treatment  and  control  schools  in  the  percentages  reported  by  principals,  although  this 
may  be  due  to  the  small  principal  sample  size.  Teachers  reported  on  use  of  these  activities  in  their  classrooms, 
ranging  from  38  percent  to  84  percent,  and  there  was  one  significant  difference  (out  of  eight  comparisons), 
with  more  treatment  teachers  than  control  teachers  reporting  the  use  of  Social  and  Emotional  Development 
activities  in  their  classrooms  (83%  versus  64%).  With  respect  to  the  use  of  schoolwide  activities,  33  percent  to 
61  percent  of  teachers  reported  that  their  schools  used  such  activities.  There  was  one  significant  difference 
here  (out  of  six  comparisons),  with  more  control  teachers  reporting  the  use  of  school  newspapers  or  bulletins 
than  treatment  teachers  (54%  versus  33%). 
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Table  7.10.  Principal  and  teacher  initial  reports  on  use  of  SACD  programs  or  activities  in 
sample — 4Rs 


SACD  program  or  activity 

Total 

Treatment 

Control 

Principal  sample  size 

14 

7 

7 

Teacher  sample  size 

177 

92 

85 

Principals  reporting  that  school  had  programs  or  activities 
to  promote  the  following  SACD  goals  (percent) 

Violence  prevention  and  peace  promotion 

92.9 

100.0 

85.7 

Social  and  emotional  development 

100.0 

100.0 

100.0 

Character  education 

100.0 

100.0 

100.0 

Tolerance  and  diversity 

100.0 

100.0 

100.0 

Risk  prevention  and  health  promotion 

92.9 

100.0 

85.7 

Civic  responsibility  and  community  service 

100.0 

100.0 

100.0 

Behavior  management 

64.3 

57.1 

71.4 

None  of  the  above 

0.0 

0.0 

0.0 

Teachers  reporting  on  using  programs  or  activities  in  their  class 
to  promote  the  following  SACD  goals  (percent) 


Violence  prevention  and  peace  promotion 

64.4 

71.9 

56.9 

Social  and  emotional  development 

73.4 

83.1* 

63.8 

Character  education 

79.3 

80.2 

78.5 

Tolerance  and  diversity 

70.6 

73.9 

67.4 

Risk  prevention  and  health  promotion 

38.0 

36.6 

39.5 

Civic  responsibility  and  community  service 

60.2 

61.7 

58.8 

Behavior  management 

84.3 

88.9 

79.8 

None  of  the  above 

1.5 

t 

t 

Teachers  reporting  schoolwide  use  of  the  following 
activities  to  promote  SACD  (percent) 

Morning  announcements  or  videos 

59.6 

49.3 

69.9 

School  assemblies 

53.7 

51.9 

55.5 

School  newspapers  or  bulletins 

43.5 

32.9* 

54.1 

Special  school  days 

32.5 

29.0 

36.0 

Special  events 

61.4 

57.0 

65.8 

Other  activities 

7.0 

6.7 

7.3 

$ Reporting  standards  not  met.  Values  suppressed  to  protect  confidentiality. 

* Treatment  group  significantly  different  from  control  group  at  the  .05  level. 

NOTE:  Weights,  which  assign  equal  weight  to  each  school  within  each  of  the  programs  and  to  each  program  across  programs,  were 
used  in  producing  the  treatment,  control,  and  overall  means.  Statistical  tests  were  conducted  using  regressions  that  included 
program  indicators  to  account  for  the  sample  design  and  adjusted  for  clustering  at  the  school  level.  Sample  size  may  differ  for  some 
outcomes  due  to  nonresponse. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Teachers  reported  using  a broad  range  of  teaching  materials  to  support  SACD  activities  (table  7.11),  including 
teacher  guides  (73%),  student  materials  (41%),  instructional  aids  (31%),  giveaways  (43%),  and  children’s 
literature  (77%).  There  was  a statistically  significant  difference  (out  of  seven  comparisons)  in  the  use  of 
teacher  guides,  with  more  treatment  teachers  reporting  their  use  than  control  teachers  (88%  versus  58%),  and 
in  the  use  of  unspecified  SACD  materials,  with  treatment  teachers  reporting  using  other  types  of  materials 
more  often  than  control  teachers  (18%  versus  6%). 

Teachers  also  reported  using  a wide  variety  of  teaching  strategies  (table  7.11).  All  teachers  reported  using  any 
of  the  20  strategies  asked  about,  and  teachers  used  an  average  of  11.5  of  the  strategies.  There  were  no 
significant  differences  between  treatment  and  control  teachers  in  the  use  of  any  of  these  specific  strategies. 
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Table  7.11.  Teacher  initial  reports  on  use  of  SACD  materials  and  classroom  strategies  in 
sample — 4Rs 


SACD  material  and  classroom  strategy 

Total 

Treatment 

Control 

Teacher  sample  size 

177 

92 

85 

Teachers  using  the  following  materials  in  conjunction  with 
social  and  character  development  activities  (percent) 

Teacher  guides  (manuals,  curricula) 

72.9 

87.5** 

58.2 

Student  materials  (workbooks,  worksheets) 

41.2 

39.6 

42.9 

Instructional  aids  (games,  software,  videos) 

30.6 

37.6 

23.7 

Giveaways  (bookmarks,  stickers) 

42.9 

38.3 

47.6 

Children’s  literature 

77.0 

81.7 

72.3 

Other  types  of  materials 

12.0 

18.2* 

5.7 

Do  not  use  any  of  the  materials  listed  above 

2.9 

t 

t 

Teachers  using  any  of  the  strategies  listed  below  to  promote 

social  and  character  development  in  the  classroom  (percent) 

100.0 

100.0 

100.0 

Number  of  strategies  (listed  below)  used  by  teachers  to  promote 

social  and  character  development  in  the  classroom  (mean) 

11.5 

11.7 

11.3 

Teachers  using  each  of  the  following  strategies  to  promote 
social  and  character  development  (percent) 

Role-playing 

59.7 

61.9 

57.5 

Cooperative  learning 

97.8 

100.0 

95.6 

Peer  group  discussions 

90.6 

94.5 

86.6 

Direct  instruction  of  social  and  character  development 

85.0 

89.1 

80.8 

Skill  training 

52.8 

48.1 

57.5 

Incorporating  social  and  character  development  into 

academic  curriculum 

77.3 

81.4 

73.1 

Parent  training 

8.5 

8.4 

8.6 

Parent/community  involvement  in  program  development  or  delivery 

20.5 

26.6 

14.3 

Mentoring 

32.6 

37.6 

27.6 

Good  behavior  notes  sent  home  daily  or  weekly 

56.6 

49.4 

63.8 

Presenting  role  models 

62.2 

63.5 

60.9 

Targeted  story  reading  or  writing  on  SACD  themes 

89.7 

95.0 

84.4 

Peer  mediation 

40.3 

40.9 

39.8 

Honor  roll  for  positive  behavior 

53.6 

45.8 

61.3 

Pledges  or  recitations  on  social  and  character  development  themes 

25.5 

24.2 

26.9 

Guided  visualization 

38.0 

33.3 

42.8 

Student-led/student-assisted  instruction 

60.6 

64.5 

56.6 

Journaling 

75.4 

79.4 

71.4 

Time  out  for  negative  behavior 

85.5 

87.7 

83.2 

Daily  or  weekly  rewards  for  positive  behavior 

87.7 

85.0 

90.5 

$ Reporting  standards  not  met.  Values  suppressed  to  protect  confidentiality. 

* Treatment  group  significantly  different  from  control  group  at  the  .05  level. 

**  Treatment  group  significantly  different  from  control  group  at  the  .01  level. 

NOTE:  Weights,  which  assign  equal  weight  to  each  school  within  each  of  the  programs  and  to  each  program  across  programs,  were 
used  in  producing  the  treatment,  control,  and  overall  means.  Statistical  tests  were  conducted  using  regressions  that  included 
program  indicators  to  account  for  the  sample  design  and  adjusted  for  clustering  at  the  school  level.  Sample  size  may  differ  for  some 
outcomes  due  to  nonresponse. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Principals  and  teachers  reported  on  participation  in  and  amount  of  SACD  training  and  staff  development 
provided  over  the  previous  12  months  (table  7.12).  Principals  reported  higher  participation  rates  (71%  versus 
70%)  and  more  hours  of  training  (10.5  versus  9.7)  than  did  teachers.  There  was  a significant  difference 
between  percentages  of  treatment  and  control  teachers  who  reported  participating  in  SACD  training,  and  this 
favored  the  treatment  group  (84%  versus  56%).  The  treatment  teachers  also  reported  significandy  more 
training  than  did  the  control  teachers  on  two  specific  SACD  activities  (out  of  seven):  Social  and  Emotional 
Development  (51%  versus  17%)  and  Character  Education  (54%  versus  20%). 


Table  7.12.  Principal  and  teacher  initial  reports  on  SACD  professional  development  in  sample — 
4Rs 


SACD  professional  development 

Total 

Treatment 

Control 

Principal  sample  size 

14 

7 

7 

Teacher  sample  size 

177 

92 

85 

Principals  reporting  that  staff  participated  in  social  and  character 
development  training  within  the  past  year  (percent) 

71.4 

100.0 

42.9 

Teachers  reporting  participation  in  social  and  character  development 
training  within  the  past  12  months  (percent) 

70.3 

84.2* 

56.3 

Number  of  hours  of  social  and  character  development  training 
principals  report  were  provided  to  each  staff  person  last  year  (mean) 

10.5 

12.1 

8.9 

Number  of  hours  of  social  and  character  development  training 
teachers  report  receiving  during  the  past  12  months  (mean) 

9.7 

12.4 

7.1 

Teachers  reporting  receiving  training  in  the  past  12  months 
in  the  following  areas  (percent) 

Violence  prevention  and  peace  promotion 

36.5 

43.2 

29.8 

Social  and  emotional  development 

34.4 

51.3** 

17.4 

Character  education 

36.9 

54.3** 

19.5 

Tolerance  and  diversity 

20.5 

28.6 

12.4 

Risk  prevention  and  health  promotion 

12.8 

14.0 

11.6 

Civic  responsibility  and  community  service 

13.8 

15.1 

12.6 

Behavior  management 

36.6 

37.7 

35.6 

* Treatment  group  significantly  different  from  control  group  at  the  .05  level. 


**  Treatment  group  significantly  different  from  control  group  at  the  .01  level. 

NOTE:  Weights,  which  assign  equal  weight  to  each  school  within  each  of  the  programs  and  to  each  program  across  programs,  were 
used  in  producing  the  treatment,  control,  and  overall  means.  Statistical  tests  were  conducted  using  regressions  that  included 
program  indicators  to  account  for  the  sample  design  and  adjusted  for  clustering  at  the  school  level.  Sample  size  may  differ  for  some 
outcomes  due  to  nonresponse. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 


The  data  on  the  initial  level  of  SACD  activity  emphasize  that  the  control  condition  was  a “standard  practice” 
control.  Standard  practice  at  the  control  schools  included  using  SACD  activities,  materials,  and  practices, 
along  with  professional  development,  at  rates  and  in  types  and  amounts  similar  to  the  treatment  schools.  For 
example,  the  percentages  of  teachers  who  reported  using  programs  or  activities  to  promote  specific  SACD 
goals  ranged  from  37  percent  to  89  percent  in  the  treatment  schools  and  from  40  percent  to  80  percent  in  the 
control  schools.  However,  the  7 significant  differences  between  the  treatment  and  control  conditions  in  the 
use  of  SACD  activities  was  more  than  expected  by  chance  (3  out  of  62  comparisons),  and  6 of  these  cases 
favored  the  treatment  group.  This  may  reflect  the  fact  that  program  implementation  and  program  training  for 
staff  started  before  initial  data  collection. 
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Impacts  on  Use  of  SACD  Activities 

The  introduction  of  the  formal  4Rs  program  would  be  expected  to  increase  the  use  of  SACD  activities  in  the 
treatment  schools  in  comparison  to  the  control  schools.  The  analysis  of  this  impact  was  based  on  the  Teacher 
Report  on  Classroom  and  School.  Every  spring,  third-,  fourth-,  and  fifth-grade  teachers  provided  information 
through  the  Teacher  Report  on  Classroom  and  School  about  the  social  and  character  development  activities 
they  used  in  their  classroom.  Specifically,  information  from  the  TRCS  was  used  to  determine  the  difference 
between  treatment  and  control  teachers  in  these  areas: 

1 . the  use  of  SACD  activities  in  their  classrooms  overall  and  by  SACD  goal; 

2.  the  use  of  materials  and  strategies  to  implement  the  SACD  activities  within  classrooms  and  within 
the  entire  school; 

3.  the  use  of  staff  development  to  support  the  teachers;  and 

4.  teacher  support  for  SACD  efforts  in  the  school  and  the  use  of  practices  conducive  to  the  social  and 
character  development  of  students. 

Teacher  Report  on  Classroom  and  School  consent  and  completion  rates  (table  7.4)  led  to  89  percent  to  98 
percent  of  all  teachers  having  data  for  the  3 years.  To  estimate  intervention  impacts  for  each  of  the  outcomes, 
testing  of  the  statistical  significance  of  the  differences  in  means  was  used.  Preliminary  analysis  indicated  little 
or  no  gains  in  precision  from  using  covariates.  Before  testing  the  mean  differences,  the  data  were  weighted 
such  that  each  school  received  equal  weight.  Standard  errors  of  the  impact  estimates  account  for  the 
clustering  of  teachers  within  schools.  In  addition,  a set  of  heuristics  was  applied  to  determine  whether  each 
outcome  domain  was  statistically  significant  after  adjustments  were  made  for  the  multiple  tests  conducted. 

Use  of  Activities 

The  percentages  of  control  teachers  who  reported  using  any  SACD  activities  in  their  classrooms  ranged  from 
76  percent  to  86  percent  over  the  3 years  (table  7.13,  panels  1 and  2).  For  the  six  individual  SACD  goals,  the 
percentages  ranged  from  47  percent  to  67  percent  in  Year  1,  44  percent  to  66  percent  in  Year  2,  and  43 
percent  to  60  percent  in  Year  3.  Control  teachers’  use  of  behavior  management  activities  ranged  from  68 
percent  to  76  percent  over  this  period.  The  percentages  of  control  teachers  who  reported  using  any  SACD 
activities  for  at  least  1 hour  per  week  in  their  classrooms  ranged  from  51  percent  to  69  percent  over  the  3 
years.  For  the  six  individual  SACD  goals,  the  percentages  ranged  from  15  percent  to  36  percent  in  Year  1,  11 
percent  to  28  percent  in  Year  2,  and  11  percent  to  31  percent  in  Year  3.  Control  teachers’  use  of  behavior 
management  activities  ranged  from  37  percent  to  54  percent  over  this  period. 

For  teachers’  reported  use  of  any  SACD  activity,  48  comparisons  were  made,  with  2 expected  to  be 
significant  by  chance.  The  percentages  of  treatment  teachers  using  any  SACD  activity  (panel  1)  were 
significantly  different  from  the  percentages  of  control  teachers  in  Year  1 (impact  =15  percentage  points)  and 
Year  3 (impact  =19  percentage  points).  Significantly  larger  percentages  of  treatment  teachers  reported  using 
activities  targeting  violence  prevention  and  peace  promotion  (impact  = 31,  19,  and  31  percentage  points), 
social  and  emotional  development  (impact  = 33,  27,  and  24  percentage  points),  character  education  (impact  = 
26,  18,  and  26  percentage  points),  and  tolerance  and  diversity  (impact  = 25,  27,  and  27  percentage  points)  in 
all  3 years.  In  Year  2,  there  was  a significant  difference  between  treatment  and  control  teachers  in  their  use  of 
civic  responsibility  and  community  service  activities,  with  more  control  teachers  reporting  their  use  (impact  = 
-16  percentage  points).  A similar  overall  pattern  was  seen  in  teachers’  reported  use  of  these  activities  for  at 
least  1 hour  per  week  (panel  2).  Treatment  teachers’  use  of  any  SACD  activity  for  at  least  1 hour  per  week 
was  significant  in  all  3 years  (impact  = 32,  17,  and  24  percentage  points).  Violence  prevention  and  peace 
promotion  was  significantly  impacted  in  Years  2 and  3 (impact  = 30  and  34  percentage  points).  Impacts  on 
social  and  emotional  development  (impact  = 31,  36,  and  25  percentage  points),  character  education  (impact  = 
29,  35,  and  29  percentage  points),  and  tolerance  and  diversity  (impact  = 30,  34,  and  30  percentage  points) 
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were  found  in  all  3 years.  Civic  responsibility  and  community  service  was  positively  significantly  impacted  in 
Year  2 (impact  =10  percentage  points).  Behavior  management  was  significandy  impacted  in  Year  3 (impact  = 
29  percentage  points).  After  the  heuristics  were  applied,  the  domain  for  engagement  in  SACD  activities 
showed  that  the  4Rs  program  had  statistically  significant  impacts  in  all  3 years. 

For  teachers’  reported  use  of  any  named  SACD  activity  (panels  3 and  4),  42  comparisons  were  made,  with  2 
expected  to  be  significant  by  chance.  Several  of  the  impact  estimates  were  statistically  significant  over  the  3 
years.  This  pattern  holds  for  the  use  of  named  activities  at  least  1 hour  per  week.  The  use  of  any  named 
activity  was  significandy  impacted  in  all  3 years,  both  overall  and  for  at  least  1 hour  per  week  (impacts  ranged 
from  40  to  49  percentage  points).  Violence  prevention  and  peace  promotion  was  significandy  impacted  in  all 
3 years,  both  overall  and  for  at  least  1 hour  per  week  (impacts  ranged  from  27  to  44  percentage  points),  as 
were  social  and  emotional  development  (impacts  ranged  from  29  to  43  percentage  points),  character 
education  (impacts  ranged  from  31  to  45  percentage  points),  and  tolerance  and  diversity  (impacts  ranged 
from  21  to  49  percentage  points).  The  use  of  risk  prevention  and  health  promotion  activities  overall  and  for 
at  least  1 hour  per  week  were  significantly  impacted  in  Year  2 (impact  =15  and  11  percentage  points).  Civic 
responsibility  and  community  service  was  significandy  impacted  in  all  3 years  (impact  = 10,  15,  and  16 
percentage  points).  The  overall  impact  of  the  4Rs  program  on  the  domain  for  engagement  in  named  SACD 
activities  was  significant  in  all  3 years. 
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Table  7.13.  Impacts  on  teacher-reported  SACD  classroom  activities — 4Rs 


Panel  1 : Engagement  in  any  activities  to  promote  SACD  goals1 

Year  1 Year  2 Year  3 

(Spring  3rd  grade) (Spring  4th  grade) (Spring  5th  grade) 

T reat-  T reat-  T reat- 


SACD  activity 

ment 

Control 

Impact 

p-value 

ment 

Control 

Impact 

p-value 

ment 

Control 

Impact 

p-value 

Teacher  sample  size 

94 

88 

91 

82 

88 

75 

Violence  prevention  and  peace  promotion 
(percent) 

85.2* 

53.9 

31.3 

0.004 

76.5* 

57.1 

19.4 

0.014 

84.6* 

53.3 

31.3 

0.000 

Social  and  emotional  development 
(percent) 

88.2* 

55.7 

32.5 

0.004 

80.2* 

52.9 

27.3 

0.010 

84.2* 

60.1 

24.1 

0.012 

Character  education  (percent) 

92.4* 

66.9 

25.5 

0.007 

83.4* 

65.6 

17.8 

0.022 

83.2* 

57.7 

25.6 

0.042 

Tolerance  and  diversity  (percent) 

80.8* 

56.0 

24.7 

0.032 

76.1** 

49.0 

27.1 

0.011 

83.5* 

56.9 

26.6 

0.001 

Risk  prevention  and  health  promotion 
(percent) 

28.7 

46.6 

-17.9 

0.174 

48.2 

44.1 

4.1 

0.550 

45.8 

42.8 

3.0 

0.696 

Civic  responsibility  and  community  service 
(percent) 

61.6 

56.6 

5.0 

0.694 

44.4* 

60.5 

-16.1 

0.050 

47.9 

57.4 

-9.5 

0.355 

Any  SACD  goal  (percent) 

97.1* 

82.1 

15.0 

0.011 

88.7 

85.6 

3.0 

0.575 

95.0* 

75.6 

19.4 

0.017 

Behavior  management  (percent) 

85.1 

73.9 

11.2 

0.196 

89. 7A 

75.9 

13.8 

0.096 

80. 7A 

67.6 

13.1 

0.087 

See  notes  at  end  of  table. 
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Table  7.13.  Impacts  on  teacher-reported  SACD  classroom  activities — 4Rs — Continued 


Panel  2:  Engagement  in  any  activities  to  promote  SACD  goals  for  at  least  1 hour  per  week 

Year  1 Year  2 Year  3 

(Spring  3rd  grade) (Spring  4th  grade) (Spring  5th  grade) 

T reat-  T reat-  T reat- 


SACD  activity 

ment 

Control 

Impact 

p-value 

ment 

Control 

Impact 

p-value 

ment 

Control 

Impact 

p-value 

Teacher  sample  size 

94 

88 

91 

82 

88 

75 

Violence  prevention  and  peace  promotion 
(percent) 

Social  and  emotional  development 

53.1 A 

27.0 

26.2 

0.053 

57.7* 

27.9 

29.8 

0.003 

55.4* 

21.8 

33.7 

0.002 

(percent) 

55.1* 

23.7 

31.4 

0.006 

63.8* 

27.8 

36.0 

0.004 

54.9* 

29.8 

25.1 

0.028 

Character  education  (percent) 

65.9* 

36.4 

29.4 

0.013 

61.0* 

26.1 

34.9 

0.000 

60.4* 

31.0 

29.3 

0.006 

Tolerance  and  diversity  (percent) 

Risk  prevention  and  health  promotion 

51.7* 

21.6 

30.1 

0.008 

56.3* 

22.7 

33.5 

0.001 

51.4* 

21.8 

29.6 

0.001 

(percent) 

Civic  responsibility  and  community  service 

5.1 

15.1 

-10.0 

0.124 

22.1 

14.9 

7.2 

0.274 

17.7 

21.0 

-3.3 

0.723 

(percent) 

25.2 

15.8 

9.3 

0.228 

20.9* 

10.7 

10.2 

0.044 

22.1 

11.1 

11.0 

0.147 

Any  SACD  goal  (percent) 

82.4* 

50.8 

31.6 

0.003 

85.7* 

69.0 

16.7 

0.023 

88.1* 

63.8 

24.3 

0.017 

Behavior  management  (percent) 

70. 4A 

53.8 

16.6 

0.090 

65.7 

47.9 

17.7 

0.114 

66.3* 

37.2 

29.1 

0.015 

See  notes  at  end  of  table. 

J3 

CD 

C/5 

c 

I— t- 

o' 


Chapter  7.  The  4Rs  Program  (Reading,  Writing,  Respect,  and 


418 


Table  7.13.  Impacts  on  teacher-reported  SACD  classroom  activities — 4Rs — Continued 

Panel  3:  Engagement  in  activities  to  promote  SACD  goals  linked  to  named  SACD  programs2 

Year  1 Year  2 Year  3 

(Spring  3rd  grade) (Spring  4th  grade) (Spring  5th  grade) 

T reat-  T reat-  T reat- 


SACD  activity 

ment 

Control 

Impact 

p-value 

ment 

Control 

Impact 

p-value 

ment 

Control 

Impact 

p-value 

Teacher  sample  size 

94 

88 

91 

82 

88 

75 

Violence  prevention  and  peace  promotion 
(percent) 

42.0* 

4.8 

37.2 

0.008 

54.0* 

12.4 

41.6 

0.000 

52.4* 

8.4 

44.0 

0.000 

Social  and  emotional  development 
(percent) 

46.6* 

3.9 

42.7 

0.010 

49.8* 

9.3 

40.5 

0.000 

52.9* 

12.1 

40.8 

0.015 

Character  education  (percent) 

47.7* 

3.9 

43.8 

0.001 

53.6* 

11.7 

41.9 

0.002 

51.1* 

6.6 

44.6 

0.004 

Tolerance  and  diversity  (percent) 

4* 

4 

31.8 

0.001 

48.4* 

6.7 

41.7 

0.000 

4* 

4 

49.2 

0.000 

Risk  prevention  and  health  promotion 
(percent) 

5.9 

8.7 

-2.9 

0.719 

23.0* 

8.0 

15.0 

0.049 

16.3 

10.3 

6.1 

0.500 

Civic  responsibility  and  community  service 
(percent) 

4* 

4 

9.6 

0.032 

18.2* 

3.3 

14.9 

0.004 

4* 

4 

15.8 

0.040 

Any  named  activity  (percent) 

60.0* 

12.4 

47.6 

0.006 

66.8* 

26.7 

40.1 

0.000 

65.3* 

21.9 

43.4 

0.013 

See  notes  at  end  of  table. 
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Table  7.13.  Impacts  on  teacher-reported  SACD  classroom  activities — 4Rs — Continued 

Panel  4:  Engagement  in  activities  to  promote  SACD  goals  linked  to  named  SACD  programs  for  at  least  1 hour  per  week 

Year  1 Year  2 Year  3 

(Spring  3rd  grade) (Spring  4th  grade) (Spring  5th  grade) 

T reat-  T reat-  T reat- 


SACD  activity 

ment 

Control 

Impact 

p-value 

ment 

Control 

Impact 

p-value 

ment 

Control 

Impact 

p-value 

Teacher  sample  size 

94 

88 

91 

82 

88 

75 

Violence  prevention  and  peace  promotion 
(percent) 

31.6* 

5.1 

26.5 

0.033 

48.8* 

9.4 

39.4 

0.000 

t* 

t 

40.4 

0.005 

Social  and  emotional  development 
(percent) 

32.8* 

4.1 

28.7 

0.034 

45.4* 

6.6 

38.8 

0.003 

43.8* 

7.3 

36.6 

0.020 

Character  education  (percent) 

35.6* 

4.2 

31.4 

0.006 

48.2* 

7.8 

40.4 

0.001 

48.3* 

3.5 

44.8 

0.001 

Tolerance  and  diversity  (percent) 

t* 

t 

21.3 

0.012 

40.0* 

3.6 

36.4 

0.002 

r 

t 

40.1 

0.001 

Risk  prevention  and  health  promotion 
(percent) 

t 

t 

-4.9 

0.264 

15.0* 

3.7 

11.4 

0.018 

10.1 

6.8 

3.3 

0.660 

Civic  responsibility  and  community  service 
(percent) 

tA 

t 

5.9 

0.097 

13.1 

0.0 

13.1 

t 

15.8 

0.0 

15.8 

t 

Any  named  activity  (percent) 

48.6* 

8.7 

40.0 

0.003 

60.1* 

16.5 

43.6 

0.000 

59.1* 

10.0 

49.1 

0.005 

t Not  applicable. 

$ Reporting  standards  not  met.  Values  suppressed  to  protect  confidentiality. 

* Treatment  group  significantly  different  from  control  group  at  the  .05  level. 

A Treatment  group  significantly  different  from  control  group  at  the  .10  to  > .05  level. 

1 1n  Year  1 at  least  half  of  the  impacts  were  positive  and  statistically  significant  and  no  impact  was  negative  and  statistically  significant  based  on  univariate  statistical  tests,  the  omnibus 
impact  for  all  the  outcomes  measured  together  was  positive  and  statistically  significant  on  the  basis  of  a multivariate  statistical  test,  and  at  least  one  outcome  remained  positive  and 
statistically  significant  and  no  outcome  was  negative  and  statistically  significant  after  applying  the  Benjamini-Hochberg  (1995)  procedure  to  adjust  significance  levels  downward  to 
account  for  the  multiple  testing  of  impacts.  In  Years  2 and  3 at  least  half  of  the  impacts  were  positive  and  statistically  significant  and  no  impact  was  negative  and  statistically  significant 
based  on  univariate  tests,  and  at  least  one  outcome  remained  positive  and  statistically  significant  and  no  outcome  was  negative  and  statistically  significant  after  applying  the 
Benjamini-Hochberg  (1995)  procedure  to  adjust  significance  levels  downward  to  account  for  the  multiple  testing  of  impacts. 

2 In  Years  1 and  3 at  least  half  of  the  impacts  were  positive  and  statistically  significant  and  no  impact  was  negative  and  statistically  significant  based  on  univariate  statistical  tests,  the 
omnibus  impact  for  all  the  outcomes  measured  together  was  positive  and  statistically  significant  on  the  basis  of  a multivariate  statistical  test,  and  at  least  one  outcome  remained 
positive  and  statistically  significant  and  no  outcome  was  negative  and  statistically  significant  after  applying  the  Benjamini-Hochberg  (1995)  procedure  to  adjust  significance  levels 
downward  to  account  for  the  multiple  testing  of  impacts.  In  Year  2 at  least  half  of  the  impacts  were  positive  and  statistically  significant  and  no  impact  was  negative  and  statistically 
significant  based  on  univariate  tests,  and  at  least  one  outcome  remained  positive  and  statistically  significant  and  no  outcome  was  negative  and  statistically  significant  after  applying  the 
Benjamini-Hochberg  (1995)  procedure  to  adjust  significance  levels  downward  to  account  for  the  multiple  testing  of  impacts. 

NOTE:  Weights,  which  assign  equal  weight  to  each  school  within  the  program,  were  used  in  producing  the  treatment,  control,  and  overall  means. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Use  of  Materials  and  Strategies 

For  use  of  materials  and  strategies  to  support  SACD  goals,  87  comparisons  were  made,  with  4 expected  to  be 
significant  by  chance.  Several  significant  impacts  were  found  on  treatment  teachers’  use  of  SACD  materials  in 
all  years  (table  7.14).  In  Year  1,  there  was  an  effect  on  the  use  of  teacher  guides  (impact  = 42  percentage 
points),  which  was  also  seen  in  Year  2 (impact  = 37  percentage  points)  and  Year  3 (impact  = 27  percentage 
points).  In  Year  2,  there  was  an  impact  on  the  use  of  student  materials  (impact  = 27  percentage  points)  and 
another  in  Year  3 (impact  =15  percentage  points).  In  Years  2 and  3,  there  was  an  impact  on  the  use  of 
children’s  literature  (impact  = 24  and  27  percentage  points).  In  Year  2,  control  teachers  reported  significandy 
greater  use  of  other  types  of  materials  (impact  = -18  percentage  points).  Regarding  use  of  instructional 
techniques,  there  was  a significant  impact  on  use  of  role-playing  in  all  3 years  (impact  = 29,  32,  and  18 
percentage  points).  In  Year  2,  there  were  significant  impacts  on  the  use  of  direct  instruction  of  SACD  (impact 
= 12  percentage  points)  and  on  incorporating  SACD  into  the  academic  curriculum  (impact  =19  percentage 
points).  Other  impacts  were  seen  on  targeted  story  reading  or  writing  (impact  =11  percentage  points)  and 
student-led/student-assisted  instruction  (impact  =18  percentage  points).  In  Year  3,  there  was  an  impact  on 
skill  training  (impact  =11  percentage  points),  guided  visualization  (impact  = 21  percentage  points),  and  the 
average  number  of  strategies  teachers  reported  (by  1.3  strategies  on  average).  The  4Rs  program  had 
significant  impacts  on  the  domain  of  materials  and  strategies  in  Years  1 and  3. 
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Table  7.14.  Impacts  on  use  of  SACD  classroom  materials  and  teaching  strategies — 4Rs 


Year  1 

(Spring  3rd  grade) 

Year  2 

(Spring  4th  grade) 

Year  3 

(Spring  5th  grade) 

SACD  material  and  teaching  strategy 

T reat- 
ment 

Control 

Impact 

p-value 

T reat- 
ment 

Control 

Impact 

p-value 

T reat- 
ment 

Control 

Impact 

p-value 

Teacher  sample  size 

94 

88 

91 

82 

88 

75 

Use  of  SACD  materials  (percent) 

Teacher  guides  (manuals,  curricula) 

88.9* 

46.9 

42.0 

0.001 

87.6* 

51.1 

36.5 

0.000 

81.2* 

54.3 

26.9 

0.002 

Student  materials  (workbooks  or  sheets) 

34.3 

39.0 

-4.7 

0.621 

62.5* 

35.1 

27.3 

0.008 

59.1* 

43.9 

15.2 

0.029 

Instructional  aids  (games,  software,  videos) 

31.3 

22.3 

8.9 

0.233 

28.0 

29.5 

-1.6 

0.825 

40. 0A 

25.7 

14.3 

0.056 

Giveaways  (bookmarks,  stickers) 

39.0 

42.7 

-3.7 

0.530 

23. 3A 

36.9 

-13.6 

0.080 

42.6 

56.2 

-13.5 

0.170 

Children’s  literature 

71.4 

57.6 

13.8 

0.117 

76.6* 

52.5 

24.0 

0.023 

81.1* 

54.0 

27.2 

0.001 

Other  types  of  materials 

9.6 

7.8 

1.7 

0.738 

3.6* 

21.7 

-18.1 

0.004 

t 

t 

3.9 

0.236 

Did  not  use  any  of  these  materials 

r 

t 

-16.0 

0.009 

6.4A 

15.4 

-8.9 

0.077 

4.9 

15.8 

-11.0 

0.100 

Use  of  teaching  strategies  (percent) 

Role-playing 

85.9* 

57.3 

28.6 

0.000 

94.4* 

62.7 

31.6 

0.000 

94.1* 

76.3 

17.9 

0.020 

Cooperative  learning 

97.6 

95.6 

1.9 

0.427 

99.0 

97.6 

1.5 

0.433 

100.0 

98.7 

1.3 

t 

Peer  group  discussions 

94.7 

91.7 

3.0 

0.492 

94.4 

89.8 

4.6 

0.324 

100.0 

96.3 

3.7 

t 

Direct  instruction  of  SACD 

89.2 

83.0 

6.2 

0.441 

92.6* 

81.0 

11.6 

0.006 

95. 3A 

83.1 

12.2 

0.077 

Skill  training 

66.2 

57.4 

8.7 

0.282 

60.7 

52.8 

7.9 

0.327 

90.4* 

79.1 

11.4 

0.032 

Incorporating  SACD  into  academic 
curriculum 

87.1 A 

71.2 

16.0 

0.063 

88.6* 

69.4 

19.1 

0.007 

96.1 

86.1 

10.1 

0.116 

Parent  training 

7.6 

3.4 

4.2 

0.278 

4.9 

7.9 

-3.0 

0.558 

25.1 

28.7 

-3.6 

0.632 

Parent/community  involvement 

21.2 

17.2 

4.0 

0.551 

23.2 

13.7 

9.4 

0.231 

47.4 

39.8 

7.6 

0.318 

Mentoring 

30.4 

28.5 

1.9 

0.817 

35.9 

34.0 

1.9 

0.866 

54.4 

49.3 

5.1 

0.591 

Good  behavior  notes  sent  home  daily 
or  weekly 

46. 5A 

65.0 

-18.6 

0.062 

55.3 

59.4 

-4.0 

0.650 

87.7 

80.5 

7.2 

0.229 

Presenting  role  models 

60.5 

65.4 

-4.9 

0.598 

63.5 

72.3 

-8.8 

0.235 

78.3 

77.5 

0.8 

0.906 

See  notes  at  end  of  table. 
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Table  7.14.  Impacts  on  use  of  SACD  classroom  materials  and  teaching  strategies — 4Rs — Continued 


Year  1 

(Spring  3rd  grade) 

Year  2 

(Spring  4th  grade) 

Year  3 

(Spring  5th  grade) 

SACD  material  and  teaching  strategy1 

T reat- 
ment 

Control 

Impact 

p-value 

T reat- 
ment 

Control 

Impact 

p-value 

T reat- 
ment 

Control 

Impact 

p-value 

Use  of  teaching  strategies  (percent) — 
Continued 

Targeted  story  reading  or  writing  on  social 
and  character  development  themes 

94.2 

85.4 

8.8 

0.113 

98.8* 

87.7 

11.1 

0.034 

96.8 

93.0 

3.7 

0.174 

Peer  mediation 

50.3A 

' 41.7 

8.6 

0.085 

58.3 

46.5 

11.8 

0.249 

73.4 

64.1 

9.3 

0.126 

Honor  roll  for  positive  behavior 

40.3 

56.3 

-15.9 

0.297 

49.0 

56.2 

-7.1 

0.633 

68.2 

69.2 

-1.0 

0.924 

Pledges  or  recitations  on  social  and 
character  development  themes 

31.9 

32.9 

-1.0 

0.905 

27.1 

30.7 

-3.6 

0.693 

57.3 

50.5 

6.8 

0.361 

Guided  visualization 

43.8 

38.7 

5.1 

0.324 

54.5 

53.2 

1.3 

0.889 

78.2* 

57.7 

20.5 

0.002 

Student-led/student-assisted  instruction 

53.7 

53.8 

-0.1 

0.993 

67.8* 

50.2 

17.6 

0.014 

82.4 

76.5 

5.9 

0.455 

Journaling 

67.4 

69.0 

-1.5 

0.865 

75. 6A 

61.5 

14.1 

0.093 

85.1 

77.4 

7.7 

0.275 

Time  out  for  negative  behavior 

89.5 

86.4 

3.1 

0.501 

79.0 

82.3 

-3.4 

0.549 

91.0 

82.2 

8.8 

0.181 

Daily  or  weekly  rewards  for  positive 
behavior 

80.6 

86.8 

-6.2 

0.439 

83.5 

82.8 

0.7 

0.901 

97.1 

95.2 

1.8 

0.550 

Any  strategy 

100.0 

99.0 

1.0 

t 

100.0 

100.0 

0.0 

t 

100.0 

98.7 

1.3 

t 

Number  of  strategies  (mean) 

12.2 

11.8 

0.3 

0.612 

1 2.9A 

11.8 

1.1 

0.073 

15.6* 

14.3 

1.3 

0.028 

t Not  applicable. 

$ Reporting  standards  not  met.  Values  suppressed  to  protect  confidentiality. 

* Treatment  group  significantly  different  from  control  group  at  the  .05  level. 

A Treatment  group  significantly  different  from  control  group  at  the  .10  to  > .05  level. 

1 1n  Years  1 and  3 and  at  least  one  outcome  remained  positive  and  statistically  significant  and  no  outcome  was  negative  and  statistically  significant  after  applying  the  Benjamini- 
Hochberg  (1995)  procedure  to  adjust  significance  levels  downward  to  account  for  the  multiple  testing  of  impacts. 

NOTE:  Weights,  which  assign  equal  weight  to  each  school  within  the  program,  were  used  in  producing  the  treatment,  control,  and  overall  means. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Regarding  the  use  of  schoolwide  strategies,  18  comparisons  were  made  between  treatment  and  control 
teacher  reports,  with  1 expected  to  be  significant  by  chance.  There  were  no  significant  impacts  on  teacher 
reports  of  schools’  use  of  specific  schoolwide  strategies  (these  data  are  not  shown  in  a table). 

Participation  in  Professional  Development 

Regarding  reported  participation  in  professional  development,  27  comparisons  were  made  over  3 years,  with 
1 expected  to  be  significant  by  chance.  In  Year  1,  the  4Rs  intervention  had  a statistically  significant  impact  on 
teachers’  participation  in  professional  development  training  (impact  = 40  percentage  points);  mean  hours  of 
SACD  training  (by  4.8  hours  on  average);  and  training  in  the  specific  goals  of  violence  prevention  and  peace 
promotion  (impact  = 25  percentage  points),  social  and  emotional  development  (impact  = 25  percentage 
points),  character  education  (impact  = 39  percentage  points),  and  tolerance  and  diversity  (impact  = 20 
percentage  points)  (table  7.15).  There  was  also  a significant  impact  on  behavior  management  training  (impact 
= 19  percentage  points).  The  impact  on  mean  hours  of  SACD  training  remained  significant  in  Year  3 (by  2.5 
hours  on  average).  The  significant  impacts  on  violence  prevention  and  peace  promotion  training  remained  in 
Year  2 (impact  =19  percentage  points)  and  Year  3 (impact  =12  percentage  points),  as  did  the  impact  on 
social  and  emotional  development  and  character  education  in  Year  2 (impact  =16  and  11  percentage  points). 
The  impact  on  tolerance  and  diversity  training  was  seen  again  in  Year  3 (impact  =17  percentage  points). 
Significant  impact  was  found  on  the  domain  in  Years  1 and  3. 
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Table  7.15.  Impacts  on  teacher-reported  SACD  professional  development — 4Rs 


SACD  professional  development1 

Year  1 

(Spring  3rd  grade) 

Year  2 

(Spring  4th  grade) 

Year  3 

(Spring  5th  grade) 

T reat- 
ment 

Control 

Impact 

p-value 

T reat- 
ment 

Control 

Impact 

p-value 

T reat- 
ment 

Control 

Impact 

p-value 

Teacher  sample  size 

94 

88 

91 

82 

88 

75 

SACD  training  in  past  12  months  (percent) 

81.6* 

41.2 

40.4 

0.000 

63. 9A 

48.6 

15.3 

0.083 

54.1 A 

30.7 

23.4 

0.052 

Hours  of  SACD  training  (mean) 

9.3* 

4.4 

4.8 

0.005 

7.4 

4.8 

2.5 

0.448 

t* 

t 

2.5 

0.037 

Training  by  goal  (percent) 

Violence  prevention  and  peace  promotion 

44.0* 

19.4 

24.6 

0.003 

36.7* 

17.4 

19.3 

0.031 

25.0* 

12.8 

12.3 

0.027 

Social  and  emotional  development 

40.8* 

15.8 

25.0 

0.003 

32.5* 

16.1 

16.4 

0.038 

22. 6A 

8.3 

14.3 

0.087 

Character  education 

50.2* 

10.9 

39.4 

0.003 

23.2* 

12.3 

10.9 

0.038 

20.1 

9.4 

10.7 

0.133 

Tolerance  and  diversity 

27.8* 

8.0 

19.8 

0.001 

23.1 

12.1 

11.1 

0.109 

t* 

t 

17.3 

0.005 

Risk  prevention  and  health  promotion 

7.1 

4.6 

2.5 

0.468 

21.0 

18.7 

2.3 

0.785 

12.8 

6.1 

6.7 

0.290 

Civic  responsibility  and  community  service 

10.2 

5.7 

4.5 

0.358 

t 

t 

6.4 

0.100 

7.3 

7.3 

0.0 

0.992 

Behavior  management 

35.0* 

15.9 

19.1 

0.042 

20.4 

18.8 

1.6 

0.774 

14.9 

13.4 

1.4 

0.796 

4:  Reporting  standards  not  met.  Values  suppressed  to  protect  confidentiality. 

* Treatment  group  significantly  different  from  control  group  at  the  .05  level. 

A Treatment  group  significantly  different  from  control  group  at  the  .10  to  > .05  level. 

1 1n  Years  1 and  3 at  least  one  outcome  remained  positive  and  statistically  significant  and  no  outcome  was  negative  and  statistically  significant  after  applying  the  Benjamini-Hochberg 
(1995)  procedure  to  adjust  significance  levels  downward  to  account  for  the  multiple  testing  of  impacts.  In  Year  1 at  least  half  of  the  impacts  were  positive  and  statistically  significant 
and  no  impact  was  negative  and  statistically  significant  based  on  univariate  statistical  tests. 

NOTE:  Weights,  which  assign  equal  weight  to  each  school  within  the  program,  were  used  in  producing  the  treatment,  control,  and  overall  means. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 


J3 

CD 

cn 

O 


Chapter  7.  The  4Rs  Program  (Reading,  Writing,  Respect,  and 


Chapter  7.  The  4Rs  Program  (Reading,  Writing,  Respect,  and  Resolution) 


Attitudes  and  Practices 

Teachers  reported  on  their  enthusiasm  for  SACD  efforts  in  their  schools  (these  data  are  not  shown  in  a table) 
by  indicating  enthusiasm,  cooperation,  or  open  dislike.  They  also  reported  on  the  SACD  practices  of  teachers 
and  staff  members  in  their  schools  (these  data  are  not  shown  in  a table).  The  practices  included  modeling 
positive  character  and  behavior  traits  with  students  and  fellow  teachers,  involving  students  in  making 
decisions,  giving  students  a voice  in  school  governance,  school  encouragement  of  parent  involvement  in 
children’s  social  and  character  development,  and  the  use  of  developmentally  appropriate  discipline  strategies 
rather  than  punishment  for  misbehavior.  Twenty-seven  comparisons  were  made  over  3 years,  with  1 expected 
to  be  significant  by  chance.  There  were  no  statistically  significant  impacts  on  teachers’  enthusiasm  for  SACD 
efforts  or  reported  SACD  practices  in  their  schools  in  any  of  the  years  (these  data  are  not  shown  in  a table). 
The  4Rs  program  had  no  impact  on  the  domain  in  any  year. 

Year-by-Year  Impacts  on  Students  and  Perceptions  of  School  Climate 

The  primary  research  question  for  the  4Rs  evaluation  was  this: 

What  is  the  average  effect  of  the  4Rs  program  on  children’s  social  and  emotional  competence, 
behavior,  academics,  and  perceptions  of  school  climate? 

The  first  approach  to  answering  this  question  was  to  examine  the  year-by-year  impacts  of  the  4Rs  program  on 
these  student  and  school  climate  outcomes  over  the  3 years  as  the  students  progressed  from  third  through 
fifth  grades. 

Equation  (2)  (described  in  chapter  1)  was  estimated  to  provide  the  impacts  of  the  4Rs  program  on  the  20 
outcomes  using  data  from  the  14  treatment  and  control  schools.  For  the  4Rs  evaluation,  equation  (2) 
excluded  the  program  fixed  effects  (Op)  and  included  program-specific  covariates  and  random  school  effects 
covariates.  Table  7.16  lists  the  covariates  used  with  outcomes  from  each  report  in  the  4Rs  analysis. 
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Table  7.16.  Covariates  used  with  outcomes  from  each  report  for  analysis — 4Rs 


Potential  covariate 

CR 

outcome 

PCR 

outcome 

TRS 

outcome 

TRCS 

outcome 

Total  number 

10 

26 

19 

6 

Child-reported 

Female 

✓ 

✓ 

Hispanic 

V 

✓ 

Black  (non-Hispanic) 

V 

✓ 

Other  ethnicity 

V 

✓ 

Age  in  years 

V 

✓ 

V 

Scales 

Afraid  at  School 

✓ 

Altruistic  Behavior 

Empathy 

✓ 

V 

Engagement  with  Learning 

V 

Negative  School  Orientation 

✓ 

Normative  Beliefs  About  Aggression 

V 

V 

Sense  of  School  as  a Community 

Problem  Behavior 

✓ 

Self-Efficacy  for  Peer  Interactions 

Victimization  at  School 

V 

Primary  caregiver-reported 

Age  in  years 

V 

✓ 

V 

Completed  high  school  or  equivalent 

✓ 

V 

Some  college 

✓ 

s 

Bachelor’s  or  higher  degree 

✓ 

V 

Highest  level  of  education  in  household 

Completed  high  school  or  equivalent 

Some  college 

Bachelor’s  or  higher  degree 

Mother  present  in  home  life 

V 

Mother  and  father  present 

V 

Respondent  someone  other  than  mother  or  father 

s 

Number  of  people  in  household 

✓ 

✓ 

Household  income:  $20,000  to  $40,000 

✓ 

Household  income:  $40,000  to  $60,000 

✓ 

Household  income:  More  than  $60,000 

✓ 

Income-to-poverty-threshold  ratio:  Below  135  percent 

✓ 

Income-to-poverty-threshold  ratio:  135-185  percent 

✓ 

Full-time  employment 

✓ 

Part-time  employment 

✓ 

See  notes  at  the  end  of  table. 
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Table  7.16.  Covariates  used  with  outcomes  from  each  report  for  analysis — 4Rs — Continued 


Potential  covariate 

CR 

outcome 

PCR 

outcome 

TRS 

outcome 

TRCS 

outcome 

Parental  scales 

APQ-Poor  Monitoring  and  Supervision  Subscale 

✓ 

APQ-Positive  Parenting  Subscale 

Child-Centered  Social  Control 

Confusion,  Hubbub,  and  Order 

✓ 

✓ 

Community  Resources 

Community  Risk 

✓ 

Parent  and  Teacher  Involvement 

Child  scales 

Altruistic  Behavior 

Positive  Social  Behavior 

✓ 

✓ 

Problem  Behavior 

✓ 

✓ 

Teacher-reported 

Female 

✓ 

Hispanic 

✓ 

Black  (non-Hispanic) 

✓ 

Other  ethnicity 

✓ 

Total  teaching  experience 

✓ 

Total  experience  in  current  school 

Regular  certificate 

Other  certificate 

Highest  degree-bachelor’s 

Child  scales 

Academic  Competence  and  Motivation 

ADHD-Related  Behavior 

✓ 

Altruistic  Behavior 

Positive  Social  Behavior 

✓ 

Problem  Behavior 

Parent  and  Teacher  Involvement 

✓ 

NOTE:  Abbreviations  are 
CR:  Child  Report 
PCR:  Primary  Caregiver  Report 
TRS:  Teacher  Report  on  Student 
TRCS:  Teacher  Report  on  Classroom  and  School 
ADHD:  Attention  deficit  hyperactivity  disorder 
APC:  Alabama  Parenting  Cuestionnaire 
■s : Covariate  used 
Blank  cell:  Covariate  not  used 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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To  assess  the  statistical  power  of  the  program -level  impact  estimates,  minimum  detectable  impacts  in  effect 
size  units  (MDES)  for  each  outcome  measure  were  calculated  for  the  4Rs  evaluation  (table  7.17).  MDES 
represent  the  smallest  impacts  in  effect  size  (standard  deviation)  units  that  can  be  detected  with  a high 
probability  (80%).  MDES  are  primarily  a function  of  study  sample  sizes,  the  degrees  of  freedom  available  for 
statistical  tests,  and  design  effects  due  to  clustering  (Schochet  2005).  For  the  14  schools  included  in  the  4Rs 
evaluation,  the  MDES  ranged  from  0.088  to  0.569  for  the  child-level  outcomes  based  on  the  Child,  Caregiver, 
and  Teacher  Report  on  Student  and  from  0.180  to  0.581  for  the  school  climate  outcomes  based  on  the 
Teacher  Report  on  Classroom  and  School.  In  general,  the  MDES  for  the  school  climate  outcomes  were  larger 
than  those  for  the  child-level  outcomes. 
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Table  7.17.  Adjusted  minimum  detectable  effect  sizes  for  impact  evaluation — 4Rs 


Outcome  measure-Report 

Year  1 

Year  2 

Year  3 

Social  and  Emotional  Competence  Domain 

Self-Efficacy  for  Peer  Interaction-CR 

0.250 

0.090 

0.090 

Normative  Beliefs  About  Aggression-CR 

0.148 

0.159 

0.275 

Empathy-CR 

0.326 

0.213 

0.180 

Behavior  Domain 

Altruistic  Behavior-CR 

0.118 

0.150 

0.183 

Altruistic  Behavior-PCR 

0.130 

0.109 

0.164 

Altruistic  Behavior-TRS 

0.323 

0.540 

0.569 

Positive  Social  Behavior-PCR 

0.104 

0.108 

0.197 

Positive  Social  Behavior-TRS 

0.268 

0.117 

0.354 

Problem  Behavior-CR 

0.148 

0.253 

0.135 

Problem  Behavior-PCR 

0.104 

0.108 

0.161 

Problem  Behavior-TRS 

0.213 

0.218 

0.404 

ADHD-Related  Behavior-TRS 

0.160 

0.226 

0.449 

Academics  Domain 

Engagement  with  Learning-CR 

0.202 

0.195 

0.169 

Academic  Competence  and  Motivation-TRS 

0.159 

0.303 

0.335 

Perceptions  of  School  Climate  Domain 

Positive  School  Orientation-CR 

0.277 

0.236 

0.294 

Negative  School  Orientation-CR 

0.088 

0.190 

0.225 

Student  Afraid  at  School-CR 

0.218 

0.235 

0.137 

Victimization  at  School-CR 

0.157 

0.116 

0.218 

Feelings  of  Safety-TRCS 

0.501 

0.542 

0.568 

Student  Support  for  Teachers-TRCS 

0.180 

0.581 

0.343 

NOTE:  Abbreviations  are 
CR:  Child  Report 
PCR:  Primary  Caregiver  Report 
TRS:  Teacher  Report  on  Student 
TRCS:  Teacher  Report  on  Classroom  and  School 
ADHD:  Attention  deficit  hyperactivity  disorder 
The  minimum  detectable  effect  (MDE)  formula  used  in  the  calculations  is  as  follows: 


MDE  = f actor (df)  * 


N 


Pi  (-  + -)  + (1  - Pi)  (—  + — ) 

\St  Sr/  \STnr  SrUr) 


where  sT  and  sc  are  the  number  of  treatment  and  comparison  schools;  nT  and  nc  are  the  average  number  of  students  per 
classroom;  p,  is  the  intraclass  correlation  (ICC)  at  the  school  level;  and  factor(df)  is  a constant  that  depends  on  the  number  of 
degrees  of  freedom  (df)  available  for  analysis  (and  is  2.802  for  the  pooled  analysis). 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Table  7.18  provides  the  estimates  of  the  4Rs  program’s  impacts  on  each  of  the  20  outcomes  over  each  of  the 
3 years  (60  impacts  in  total,  with  3 expected  to  be  statistically  significant  by  chance).  Of  the  60  results,  1 was 
statistically  significant.  There  was  a detrimental  impact  on  Academic  Competence  and  Motivation  (Teacher 
Report  on  Student,  effect  size  [ES]  = -0.17)  in  Year  1.  In  addition,  there  were  substantively  important  but 
nonsignificant  detrimental  impacts  on  Feelings  of  Safety  (Teacher  Report  on  Classroom  and  School,  ES  = 
-0.42)  and  Student  Support  for  Teachers  (Teacher  Report  on  Classroom  and  School,  ES  = -0.35)  in  Year  3. 
Application  of  the  heuristics  to  adjust  for  multiple  comparisons  within  each  outcome  domain  indicated  that 
4Rs  had  a statistically  significant  detrimental  effect  on  the  Social  and  Emotional  Competence  domain  in  Year 
3 and  on  the  Behavior  domain  in  Year  1. 
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Table  7.18.  Impacts  on  child  and  school  outcomes — 4Rs 


Year  1 

(Spring  3rd  grade) 

Year  2 

(Spring  4th  grade) 

Year  3 

(Spring  5th  grade) 

Scale-Report 

T reat- 
ment 

Control 

Effect 

size 

p-value 

T reat- 
ment 

Control 

Effect 

size  p-value 

T reat- 
ment 

Control 

Effect 

size 

p-value 

Social  and  Emotional  Competence  Domain1 

Self-Efficacy  for  Peer  Interactions-CR  (+) 

2.96 

3.04 

-0.13 

0.359 

3.10 

3.12 

-0.04 

0.598 

3.15 

3.21 

-0.08 

0.338 

Normative  Beliefs  About  Aggression-CR  (-) 

1.39 

1.44 

-0.09 

0.324 

1.47 

1.49 

-0.03 

0.778 

1.59 

1.65 

-0.09 

0.545 

Empathy-CR  (+) 

2.24 

2.27 

-0.07 

0.674 

2.08 

2.10 

-0.05 

0.701 

1.95 

2.00 

-0.10 

0.382 

Behavior  Domain2 

Altruistic  Behavior-CR  (+) 

1.30 

1.40 

-0.13 

0.137 

1.14 

1.22 

-0.09 

0.365 

1.18 

1.18 

0.00 

0.985 

Altruistic  Behavior-PCR  (+) 

2.38 

2.38 

0.00 

0.989 

2.36 

2.43 

-0.11 

0.270 

2.33 

2.43 

-0.13 

0.375 

Altruistic  Behavior-TRS  (+) 

1.48 

1.58 

-0.17 

0.262 

1.48 

1.53 

-0.10 

0.730 

1.62 

1.55 

0.11 

0.698 

Positive  Social  Behavior-PCR  (+) 

2.97 

2.98 

-0.03 

0.756 

3.02 

3.04 

-0.03 

0.757 

3.11 

3.05 

0.13 

0.422 

Positive  Social  Behavior-TRS  (+) 

2.81 

2.78 

0.03 

0.788 

2.84 

2.90 

-0.08 

0.310 

2.81 

2.81 

-0.01 

0.969 

Problem  Behavior-CR  (-) 

0.40 

0.43 

-0.04 

0.654 

0.48 

0.51 

-0.05 

0.718 

0.61 

0.59 

0.03 

0.709 

Problem  Behavior-PCR  (-) 

1.55 

1.51 

0.11 

0.223 

1.51 

1.53 

-0.04 

0.720 

1.52 

1.53 

-0.01 

0.916 

Problem  Behavior-TRS  (-) 

1.60 

1.55 

0.08 

0.439 

1.51 

1.55 

-0.07 

0.516 

1.63 

1.65 

-0.04 

0.830 

ADHD-Related  Behavior-TRS  (-) 

1.84 

1.83 

0.01 

0.865 

1.69 

1.78 

-0.15 

0.205 

1.74 

1.87 

-0.20 

0.357 

Academics  Domain 

Engagement  with  Learning-CR  (+) 

3.63 

3.67 

-0.08 

0.541 

3.63 

3.64 

-0.03 

0.815 

3.63 

3.59 

0.07 

0.508 

Academic  Competence  and  Motivation- 

TRS  (+) 

2.69" 

1 2.84 

-0.17 

0.032 

2.80 

2.88 

-0.10 

0.513 

2.73 

2.89 

-0.16 

0.309 

See  notes  at  end  of  table. 
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Table  7.18.  Impacts  on  child  and  school  outcomes — 4Rs — Continued 


Year  1 

(Spring  3rd  grade) 

Year  2 

(Spring  4th  grade) 

Year  3 

(Spring  5th  grade) 

Scale-Report 

T reat- 
ment 

Control 

Effect 

size 

p-value 

T reat- 
ment 

Control 

Effect 

size 

p-value 

T reat- 
ment 

Control 

Effect 

size 

p-value 

Perceptions  of  School  Climate  Domain 
Positive  School  Orientation-CR  (+) 

2.70 

2.64 

0.08 

0.577 

2.39 

2.38 

0.01 

0.950 

2.31 

2.36 

-0.08 

0.617 

Negative  School  Orientation-CR  (-) 

2.04 

2.07 

-0.06 

0.439 

2.20 

2.19 

0.02 

0.841 

2.26 

2.25 

0.02 

0.862 

Victimization  at  School-CR  (-) 

0.83 

0.89 

-0.08 

0.436 

0.74A 

0.87 

-0.17 

0.069 

0.75 

0.78 

-0.04 

0.785 

Feelings  of  Safety-TRCS  (+) 

3.15 

3.22 

-0.09 

0.606 

3.08 

3.12 

-0.05 

0.817 

2.93 

3.27 

-0.42° 

0.146 

Student  Support  for  Teachers-TRCS  (+) 

3.10 

3.05 

0.06 

0.568 

3.14 

3.12 

0.02 

0.926 

3.08 

3.30 

-0.35° 

0.109 

* Treatment  group  significantly  different  from  control  group  at  the  .05  level. 

A Treatment  group  significantly  different  from  control  group  at  the  .10  to  > .05  level. 
a Substantive  (but  nonsignificant  at  .05  level)  effect  size  of  s .25  or  < -.25. 

1 1mpact  on  domain  found  statistically  significant  and  detrimental  in  Year  3 based  on  the  fourth  heuristic  in  which  the  statistical  model  used  to  estimate  impacts  on  the  individual 
outcomes  was  re-estimated  using  a composite  of  all  the  outcome  variables  under  a domain.  The  domain  was  found  significant  if  the  impact  on  the  composite  was  significant.  The 
composite  was  formed  by  standardizing  each  outcome  variable  using  its  standard  deviation,  combining  the  values  of  the  outcome  variables,  and  taking  the  average  of  the  final  value. 
2 Impact  on  domain  found  statistically  significant  and  detrimental  in  Year  1 based  on  the  fourth  heuristic  in  which  the  statistical  model  used  to  estimate  impacts  on  the  individual 
outcomes  was  re-estimated  using  a composite  of  all  the  outcome  variables  under  a domain.  The  domain  was  found  significant  if  the  impact  on  the  composite  was  significant.  The 
composite  was  formed  by  standardizing  each  outcome  variable  using  its  standard  deviation,  combining  the  values  of  the  outcome  variables,  and  taking  the  average  of  the  final  value. 
NOTE:  Abbreviations  are 
CR:  Child  Report 
PCR:  Primary  Caregiver  Report 
TRS:  Teacher  Report  on  Student 
TRCS:  Teacher  Report  on  Classroom  and  School 
ADHD:  Attention  deficit  hyperactivity  disorder 

The  +/-  signs  in  parentheses  indicate  the  direction  of  a beneficial  outcome.  All  impact  estimates  were  calculated  using  regression  models  where  each  program  and  school  within  a 
program  was  weighted  equally.  The  standard  errors  of  all  estimates  account  for  design  effects  due  to  unequal  weighting  and  the  clustering  of  students  within  schools.  See  table  1 .5 
for  information  about  the  measures  used  to  create  the  outcome  variables.  The  effect  size  was  calculated  by  dividing  the  estimated  impact  by  the  standard  deviation  of  the  outcome 
measure  for  the  control  group.  The  number  of  results  found  significant  was  no  more  than  expected  by  chance. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Impacts  on  Child  Outcomes  Over  Time 

The  4Rs  program’s  impacts  on  the  child  outcomes  over  time  were  estimated  using  growth  curve  models  by 
examining  treatment  and  control  group  differences  in  the  trajectories  of  student  outcomes  during  the  follow- 
up period  while  accounting  for  clustering  at  the  school  level.  The  growth  curve  models  are  estimated  using  a 
three-level  hierarchical  linear  model,  where  Level  1 corresponds  to  time,  Level  2 to  students,  and  Level  3 to 
schools  (described  in  chapter  1). 

Table  7.19  provides  the  estimates  of  the  4Rs  program’s  impacts  on  the  growth  in  student  outcomes  over  the 
3 years.  The  estimated  impacts  ranged  in  effect  size  units  (absolute  value)  from  0.00  to  0.10.  None  of  the  18 
estimated  4Rs  intervention  impacts  on  the  trajectories  of  child  outcomes  was  statistically  significandy 
different  from  zero  at  the  5 percent  level  (1  in  18  would  be  expected  to  be  significant  by  chance). 
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Table  7.19.  Impacts  on  growth  of  child  outcomes — 4Rs 


Scale-Report 

Average  growth  in  the  score  per  year1 

Mean  score  at 
implementation2 

Treatment 

Control 

Impact  on 
growth3 

Effect 

4 

size 

Standard 
error  of 
impact 

p-value  of 
impact 

Social  and  Emotional  Competence  Domain 

Self-Efficacy  for  Peer  Interactions-CR  (+) 

2.89 

0.13 

0.14 

-0.01 

-0.01 

0.03 

0.850 

Normative  Beliefs  About  Aggression-CR  (-) 

1.31 

0.13 

0.13 

-0.01 

-0.01 

0.04 

0.840 

Empathy-CR  (+) 

2.37 

-0.18 

-0.17 

-0.01 

-0.02 

0.03 

0.756 

Behavior  Domain 

Altruistic  Behavior-CR  (+) 

1.51 

-0.16 

-0.19 

0.02 

0.03 

0.06 

0.681 

Altruistic  Behavior-PCR  (+) 

2.39 

-0.02 

0.01 

-0.03 

-0.04 

0.04 

0.405 

Altruistic  Behavior-TRS  (+) 

1.46 

0.07 

0.00 

0.07 

0.10 

0.10 

0.475 

Positive  Social  Behavior-PCR  (+) 

2.94 

0.02 

0.04 

-0.02 

-0.04 

0.02 

0.400 

Positive  Social  Behavior-TRS  (+) 

2.85 

-0.05 

0.00 

-0.05 

-0.06 

0.05 

0.356 

Problem  Behavior-CR  (-) 

0.36 

0.1 2A 

0.09 

0.03 

0.04 

0.02 

0.067 

Problem  Behavior-PCR  (-) 

1.54 

0.00 

-0.02 

0.02 

0.04 

0.02 

0.391 

Problem  Behavior-TRS  (-) 

1.47 

0.06 

0.11 

-0.04 

-0.09 

0.05 

0.361 

ADHD-Related  Behavior-TRS  (-) 

1.77 

0.01 

0.05 

-0.04 

-0.05 

0.05 

0.464 

Academics  Domain 

Engagement  with  Learning-CR  (+) 

3.65 

0.01 

-0.04 

0.05 

0.07 

0.03 

0.136 

Academic  Competence  and  Motivation-TRS  (+) 

2.61 

0.05 

0.09 

-0.04 

-0.04 

0.06 

0.548 

See  notes  at  end  of  table. 
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Table  7.19.  Impacts  on  growth  of  child  outcomes — 4Rs — Continued 


Average  growth  in  the  score  per  year1 


Scale-Report 

Mean  score  at 
implementation2 

T reatment 

Control 

Impact  on 
growth3 

Effect 

size4 

Standard 
error  of 
impact 

p-value  of 
impact 

Perceptions  of  School  Climate  Domain 
Positive  School  Orientation-CR  (+) 

2.88 

-0.23 

-0.25 

0.03 

0.03 

0.05 

0.571 

Negative  School  Orientation-CR  (-) 

2.05 

0.08 

0.09 

0.00 

0.00 

0.05 

0.947 

Student  Afraid  at  School-CR  (-) 

2.59 

-0.14 

-0.15 

0.02 

0.02 

0.05 

0.716 

Victimization  at  School-CR  (-) 

0.88 

-0.06 

-0.06 

0.00 

0.00 

0.05 

0.930 

A Treatment  group  significantly  different  from  control  group  at  the  .05  level. 

1 Pertains  to  the  estimated  slope  of  the  outcome  for  the  treatment  or  control  groups. 

2 The  average  score  at  implementation  is  calculated  across  treatment  and  control  groups,  using  regression  models  for  adjustment  on  covariates. 

3 Estimated  difference  between  the  slope  of  the  treatment  and  control  groups. 

4 Effect  size:  the  slope  of  the  treatment  group  minus  the  slope  of  the  control  group  divided  by  the  standard  deviation  of  the  outcome  for  the  program's  control  group  (the  standard 
deviation  is  calculated  without  accounting  for  school-level  clustering  or  regression  adjustments). 

NOTE:  Abbreviations  are 
CR:  Child  Report 
PCR:  Primary  Caregiver  Report 
TRS:  Teacher  Report  on  Student 
ADHD:  Attention  deficit  hyperactivity  disorder 

No  findings  were  found  statistically  significant  at  or  below  the  .05  level.  The  +/-  signs  in  parentheses  indicate  the  direction  of  a beneficial  outcome.  All  impact  estimates  were 
calculated  using  HLM  6.06.  Sample  weights  were  used  in  all  analyses  to  give  (1)  each  school  equal  weight  in  each  program  (within  each  time  period)  and  (2)  each  time  period  equal 
weight  within  the  analysis.  See  table  1 .5  for  information  about  the  measures  used  to  create  the  outcome  variables. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Summary 

As  part  of  the  Social  and  Character  Development  (SACD)  initiative,  researchers  at  the  New  York  site 
evaluated  the  4Rs  program.  This  program  was  designed  to  integrate  the  teaching  and  promoting  of  social  and 
emotional  skills  and  conflict  resolution  into  the  language  arts  curriculum  and  to  create  a caring  classroom 
community.  Fourteen  public  schools  in  New  York  City  were  recruited  by  the  New  York  research  team  and 
randomly  assigned  to  treatment  and  control  conditions  to  determine  the  impact  of  4Rs  on  social  and 
character  development  activities  in  the  schools  and  on  the  child  outcome  domains  of  Social  and  Emotional 
Competence,  Behavior,  Academics,  and  Perceptions  of  School  Climate. 

Analyses  of  the  initial  characteristics  of  the  sample  (students,  caregivers,  communities,  teachers,  and  schools) 
indicated  that  randomization  to  treatment  and  control  status  produced  groups  that  were  relatively  similar  at 
the  start  of  the  study  (with  1 out  of  84  comparisons  statistically  significantly  different,  fewer  than  would  be 
expected  by  chance).  The  data  on  the  initial  level  of  SACD  activity  led  to  two  findings.  First,  treatment 
teachers  reported  greater  use  of  and  training  in  SACD  activities  than  control  teachers,  and  they  did  so  more 
often  than  would  be  expected  by  chance  (7  out  of  62  comparisons,  with  3 expected  significant  by  chance). 
There  are  two  potential  causes  for  this  finding,  and  the  analysis  cannot  be  used  to  determine  whether  the 
reason  for  such  a difference  was  that  the  two  groups  did  differ  on  their  initial  use  of  SACD  activities  (i.e.,  that 
randomization  did  not  create  similar  treatment  and  control  groups)  or  whether  the  fact  that  the  training  of  all 
treatment  teachers  and  the  implementation  of  the  4Rs  program  began  before  the  initial  data  were  collected 
(by  4 weeks)  influenced  the  teacher  reports.  Because  it  is  likely  (though  unproven)  that  the  training  and 
implementation  affected  the  teacher  reports,  these  data  were  not  considered  appropriate  for  use  as  a baseline 
measure  of  SACD  activities  and  training  in  the  treatment  schools. 

Second,  these  data  indicate  that  the  control  condition  for  the  SACD  project  was  not  a “no  treatment”  control 
but  a “standard  practice”  control.  Because  the  control  teachers  were  not  affected  by  the  implementation  of 
the  SACD  programs  before  data  collection,  their  reports  reflect  standard  practice  in  the  control  schools. 
Standard  practice  at  the  control  schools  included  reports  of  40  percent  to  80  percent  of  teachers  using  any 
SACD  activities,  96  percent  of  teachers  using  specific  materials  in  conjunction  with  these  activities,  100 
percent  using  at  least  one  of  the  specified  instructional  strategies,  and  56  percent  participating  in  SACD 
training  over  the  past  12  months. 

Analyses  of  the  impacts  of  the  4Rs  program  on  the  level  of  SACD  activities  in  the  schools  revealed  impacts 
on  the  use  of  such  activities  (66  out  of  90,  1 negative)  and  related  materials  and  strategies  (18  out  of  87,  1 
negative)  across  the  3 years.  Analyses  also  showed  more  use  of  professional  development  activities  for 
treatment  teachers  in  all  years  (13  out  of  27). 

Of  the  20  child-level  outcomes  representing  the  four  domains  of  Social  and  Emotional  Competence, 
Behavior,  Academics,  and  Perceptions  of  School  Climate  assessed  in  each  of  the  3 years  of  the  study  (a  total 
of  60  results),  1 showed  a statistically  significant  detrimental  impact  of  the  4Rs  intervention.  A growth  curve 
analysis  was  used  to  analyze  the  change  over  time  in  these  same  outcomes  between  initial  data  collection  and 
the  final  outcome  data  collection  at  the  end  of  the  study.  None  of  the  18  child-level  outcomes  assessed 
showed  a significant  impact  of  the  4Rs  program. 

The  SACD  evaluation  did  not  find  evidence  to  support  the  hypothesis  that  4Rs  had  beneficial  impacts  on 
students’  social  and  character  development.  Such  results  could  be  caused  by  the  inability  of  the  program  to 
cause  such  change,  possibly  because  the  theory  of  action  for  the  program  is  incomplete  or  the  activities  to 
carry  out  that  theory  are  not  effective. 

However,  these  results  may  also  be  due  to  the  inability  of  the  evaluation  to  observe  such  a change  due  to  the 
control  condition,  the  level  of  nonparticipation,  or  the  sample  size.  The  control  schools  continued  using  their 
standard  SACD  activities,  and  these  turned  out  to  be  high  in  quantity  and  broad  in  scope.  While  4Rs  had  a 
significant  positive  impact  on  the  amount  and  type  of  SACD  activities,  the  resulting  difference  in  the  amount 
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of  SACD  activities  between  the  treatment  and  control  schools  may  not  have  been  large  enough  to  cause 
significant  differences  in  the  student  outcomes.  Second,  38  percent  to  45  percent  of  the  students  in  the 
sample  universe  did  not  take  part  (depending  on  year)  because  of  nonconsent  or  noncompletion  of  the 
surveys.  As  a determination  could  not  be  made  as  to  whether  the  students  not  taking  part  significantly 
differed  from  those  who  did  take  part,  the  evaluation’s  results  are  valid  only  for  the  students  who  took  part.  If 
the  students  not  taking  part  were  different,  and  if  they  would  have  responded  better  to  4Rs  than  to  the  SACD 
activities  occurring  in  the  control  schools,  then  the  evaluation  could  have  underestimated  the  program’s 
impact.  Third,  the  sample  size  of  14  schools  and  the  resulting  higher  MDES  compared  to  those  for  the 
multiprogram  evaluation  may  have  reduced  the  likelihood  of  detecting  statistically  significant  effects. 
However,  it  should  be  noted  that  68  percent  of  the  MDES  for  the  60  outcomes  used  in  the  year -by-year 
analysis  were  below  0.25  (50%  were  below  0.20).  In  addition,  only  2 of  the  60  outcomes  were  found  to  be 
substantively  important,  and  they  were  detrimental. 
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Chapter  8.  Second  Step 


University  of  Maryland 
(Maryland  Site) 


Intervention 

Researchers  at  the  University  of  Maryland  (Maryland  site)  evaluated  the  Second  Step  (SS)  program.  This 
program,  one  of  several  disseminated  by  the  Committee  for  Children,  is  designed  to  promote  social 
competence  and  reduce  children’s  social  and  emotional  problems.  Table  8.1  describes  the  SS  program’s 
general  characteristics  (panel  1),  the  types  of  instruction  and  strategies  used  (panel  2),  the  professional 
development  provided  for  those  implementing  the  program  (panel  3),  and  the  social  and  character 
development  activities  (panel  4)  and  outcomes  (panel  5).  SS  includes  the  following  components: 

• The  SS  curriculum  focuses  on  developing  children’s  social  and  emotional  skills  in  the  areas  of 
empathy,  impulse  control  and  problem  solving,  and  anger  management.  Through  storytelling, 
discussion,  and  role-playing,  teachers  deliver  30-minute  classroom  lessons  to  students,  1 to  2 days  per 
week. 

• Teachers  learn  to  use  classroom  management  practices  and  to  model  social-emotional  skills  to  foster 
a positive  classroom  climate.  Teachers  give  students  opportunities  to  practice  social-emotional  skills 
in  new  situations;  positively  reinforce  students’  skill  use;  and  utilize  teachable  moments  to  provide 
coaching,  constructive  feedback,  and  positive  reinforcement  to  students  to  support  skills  used  during 
real-life  situations.  Visual  cues  are  utilized  to  illustrate  aspects  of  social  problem  solving. 

• SS  extends  into  the  home  through  newsletters,  a family  guide  and  video,  and  encouragement  of 
parent  involvement  in  helping  children  complete  their  SS  homework. 

• In  this  evaluation,  treatment  schools  established  Character  Development  Planning  Teams  (CDPTs), 
made  up  of  counselors,  the  principal,  and  other  school  staff  members  (e.g.,  teachers,  administrative 
or  office  personnel)  who  made  decisions  about  program  implementation  and  other  social  and 
character  development  activities. 
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Table  8.1 . Second  Step 

Panel  1 : General  characteristics 

Target  population 
Universal 

Program  components 
Peer:  In  class 
Parent:  Contact 

Classroom : Lessons  and  transfer  of  training 
Schoolwide:  Program  artifacts  and  modeling 
Community:  None  or  not  major  focus 
Training:  Pretraining  and  ongoing 

Level  of  integration 

Add-on  curriculum 

Flexibility 

Manualized:  Curriculum  guidebook 
Adaptability:  Character  development  planning  teams 
See  notes  at  end  of  table. 


Panel  2:  Description  of  instruction  and  strategies 

Classroom 

Lessons 

Who  delivers:  T eacher 

Activities  and  tools:  Storytelling,  discussion,  videos,  role-playing,  anticipation,  recall,  learning  points 
Content:  Empathy,  impulse  control,  problem  solving,  anger  management 
Frequency:  30-minute  lessons,  1 to  2 days  per  week 
Strategies 

Who  delivers:  T eacher 

Activities  and  tools:  Modeling  and  reinforcement  of  skills 
Frequency:  Weekly  lessons,  daily  generalization 

Supplement  to  classroom 

Take-home  letters,  homework,  family  guide,  and  video 
Schoolwide  activities 

Program  artifacts,  training  of  school  staff 

See  notes  at  end  of  table. 
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Table  8.1 . Second  Step— Continued 

Panel  3:  Professional  development 

Pre-implementation 

Teachers 

Content:  Training  on  child  development,  social  skills,  and  curriculum  delivery 
Duration:  2 days 
Other 

Content:  Character  Development  Planning  Team,  school  counselor,  and  other  staff;  training  on  child 
development,  social  skills,  and  curriculum  content 
Duration:  2V6  hours 

Ongoing  consultation 
Teachers 

Content:  Year-end  workshop  to  review  student  outcome  data  and  plan  for  following  year;  ongoing  program 
implementation  support  by  school  counselors 
Duration:  1 day 
Other 

Content:  School  counselor  guidance  on  strategies  and  challenges;  principal  and  staff  review  student 
outcome  data  and  plan  for  following  year 

Duration:  Vinjay  counselor  meetings  about  monthly;  1-day  year-end  workshop 

See  notes  at  end  of  table. 


Panel  4:  Activities  for  SACD  goals 


Violence  prevention  and  peace  promotion 

✓ 

Risk  prevention  and  health  promotion 

Social  and  emotional  development 

✓ 

Civic  responsibility  and  community  service 

Character  education 

Behavior  management 

Tolerance  and  diversity 

See  notes  at  end  of  table. 


Panel  5:  SACD  outcomes  addressed 


Engagement  with  Learning 

Academic  Competence  and  Motivation 

Altruistic  Behavior 

Positive  Social  Behavior 

Problem  Behavior 

Self-Efficacy  for  Peer  Interactions 

Normative  Beliefs  About  Aggression 


✓ 

✓ 


Empathy 

Positive  School  Orientation 
Negative  School  Orientation 
Student  Afraid  at  School 
Victimization  at  School 
Feelings  of  Safety 
Student  Support  for  Teachers 


✓ 


NOTE:  Abbreviations  are 
■/:  Activity  or  outcome  addressed 
Blank  cell:  Activity  or  outcome  not  addressed 
SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Sample  and  Random  Assignment 

The  Maryland  research  team  recruited  a total  of  12  public  elementary  schools  in  a single  school  district  in 
Maryland.  The  12  schools  were  randomly  assigned  to  treatment  and  control  conditions  prior  to  the  fall  2004 
data  collection.  A two-step  process  was  used.  First,  a pairwise  matching  algorithm  developed  by  Mathematica 
Policy  Research,  Inc.  (MPR)  was  used  to  identify  the  best  pairwise  matches  across  the  12  schools  based  on 
variables  identified  by  the  Maryland  research  team.  The  variables  used  in  the  pairwise  matching  for  the 
Maryland  site  included  the  following:  (a)  school  mobility  rate,  (b)  average  daily  attendance  rate,  (c)  average 
national  percentile  rank  of  reading  and  math  scores  for  fifth-graders,  (d)  average  national  percentile  rank  of 
reading  and  math  scores  for  third-graders,  (e)  percentage  of  students  eligible  for  free  and  reduced-price  lunch, 
(f)  school  enrollment  size,  (g)  percentage  of  families  in  the  community  with  children  in  female-headed 
households  with  no  husband  present,  (h)  percentage  of  persons  in  the  community  aged  25  years  and  older 
with  bachelor’s  degrees,  and  (i)  population  density.  Second,  using  a computer-based  pseudo  random  number 
generator,  1 school  in  each  matched  pair  was  assigned  to  either  the  intervention  or  the  control  condition.  Six 
schools  received  the  SS  program  and  6 schools  acted  as  control  schools  and  continued  to  implement  the 
social  and  character  development  activities  that  constituted  their  standard  practice.  Assignment  to  treatment 
or  control  condition  was  at  the  school  level  and  therefore  limited  the  risk  of  contamination  between 
treatment  and  control  classrooms. 

The  original  student  sample  (the  cohort  of  students  in  the  third  grade  in  the  12  schools  in  fall  2004) 
numbered  944  students  (524  treatment  and  420  control).  Table  8.2  documents  the  change  in  the  sample  over 
the  three  spring  follow-up  data  collection  periods.  Over  time,  new  entrants  to  the  cohort  became  a larger 
percentage  of  the  sample,  eventually  making  up  14  percent  of  the  sample  by  the  spring  of  Year  3.  There  were 
no  statistically  significant  differences  between  the  treatment  and  control  groups  in  the  numbers  of  new 
entrants.  The  percentage  of  the  sample  made  up  of  the  original  cohort  further  declined  as  students  left  the 
schools.  By  Year  3,  approximately  9 percent  of  the  original  sample  had  left.  There  was  a statistically 
significant  difference  between  treatment  and  control  schools  in  the  percentage  of  students  who  had  left  (i.e., 
“leavers”)  in  Year  3;  the  treatment  schools  had  a larger  percentage  of  leavers  than  control  schools  (11% 
versus  8%). 
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Table  8.2.  Sample — SS 


Year  1 Year  1 Year  2 Year  3 

(Fall  3rd  grade) (Spring  3rd  grade)  (Spring  4th  grade)  (Spring  5th  grade) 


Treat-  Treat-  Treat-  Treat- 


Characteristic 

Total 

ment 

Control 

Total 

ment 

Control 

Total 

ment 

Control 

Total 

ment 

Control 

School  sample  size 

12 

6 

6 

12 

6 

6 

12 

6 

6 

12 

6 

6 

Student  sample  size 

944 

524 

420 

951 

528 

423 

959 

533 

426 

989 

533 

456 

Stayers 

t 

t 

T 

918 

510 

408 

826 

456 

370 

855 

467 

388 

New  entrants 

t 

t 

t 

33 

18 

15 

133 

77 

56 

134 

66 

68 

New  entrants  as  a percent  of 
spring  enrollment 

t 

t 

t 

3.5 

3.4 

3.5 

13.9 

14.4 

13.1 

13.5 

12.4 

14.9 

Total  leavers  (from  original  cohort) 

t 

t 

t 

26 

14 

12 

118 

68 

50 

89 

57 

32 

Leavers  as  a percent  of  fall  2004 
enrollment 

T 

t 

t 

2.8 

2.7 

2.9 

12.5 

13.0 

11.9 

9.4 

10.9* 

7.6 

Number  of  students  per  school 

(mean) 

79 

87 

70 

79 

88 

71 

80 

89 

71 

82 

89 

76 

Range  of  number  of  students  per 

school 

46-123 

55-123 

46-102 

47-127 

54-127 

47-104 

39-125 

54-25 

39-110 

41-121 

52-121 

41-115 

f Not  applicable. 

* Treatment  group  significantly  different  from  control  group  at  the  .05  level. 
SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Implementation 

Training 

The  intervention  teachers  received  2 days  of  program  implementation  training  prior  to  the  beginning  of  the 
school  year  (table  8.1,  panel  3).  School  counselors  provided  ongoing  program  implementation  support  to 
teachers  throughout  the  school  year.  School  counselors  received  ongoing  guidance  through  half-day  monthly 
project  meetings  from  the  Maryland  research  team.  In  these  monthly  meetings,  the  research  team  provided 
detailed  implementation  feedback  to  the  school  counselors,  who  then  conveyed  the  information  to 
participating  teachers  on  an  as -needed  basis. 

Teachers  participated  in  a 2-day  training  that  focused  on  child  development,  social  skills,  and  program 
implementation.  Members  of  the  Character  Development  Planning  Teams  (CDPTs)  and  other  school  staff 
members  (e.g.,  principal,  school  counselors)  received  a condensed,  2.5-hour  version  of  the  teacher  training. 
During  the  school  year,  school  counselors  participated  in  monthly  half-day  meetings  with  the  Maryland 
research  team  to  discuss  program  implementation.  Teachers  and  CDPT  members  were  invited  to  participate 
in  a 1 -day  end-of-the-year  workshop  to  review  data  on  student  progress  and  to  plan  implementation  activities 
for  the  following  year.  Teams  opted  to  do  this  after  the  first  year  of  implementation  only. 

Data  Collection 

MPR  collected  the  multiprogram  child,  teacher,  and  school  data  for  the  Maryland  site.  Table  8.3  shows  the 
school  year  milestones  and  dates  of  implementation  for  the  Maryland  site.  Data  were  collected  in  the  fall  and 
spring  of  the  first  2 years  and  in  the  spring  of  Year  3.  The  fall  2004  multiprogram  data  collection  began  on 
October  1 1 , 2004,  and  ended  on  October  29,  2004.  The  average  time  frame  from  the  beginning  of  program 
implementation  to  the  beginning  of  fall  data  collection  was  5 weeks.  As  a result,  initial  data  collection  took 
place  after  implementation  of  the  SS  program  began.  Therefore,  these  data  provide  a measure  of  the  students, 
teachers,  and  schools  near  the  beginning  of  the  school  year,  at  a time  when  the  SS  program  had  been 
operating  for  a relatively  short  period  of  time.  The  spring  data  collection  window  was  from  April  25,  2005,  to 
May  13,  2005.  The  SS  program  had  been  implemented  for  33  weeks  at  the  time  of  the  spring  data  collection 
and  for  25  weeks  from  the  end  of  the  fall  data  collection.  Year  2 followed  a similar  pattern,  with 
implementation  occurring  at  the  start  of  the  school  year,  fall  data  collection  occurring  8 weeks  later,  and 
spring  data  collection  occurring  23  weeks  after  fall  data  collection  (and  34  weeks  after  the  start  of 
implementation).  In  spring  2007,  data  collection  occurred  34  weeks  after  the  start  of  implementation.  Data 
collection  took  from  3 to  5 weeks  at  each  collection  point. 
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Table  8.3.  Data  collection  dates — SS 


Year  1 Year  1 Year  2 Year  2 Year  3 

Data  collection  schedule  (Fall  3rd  grade)  (Spring  3rd  grade)  (Fall  4th  grade)  (Spring  4th  grade)  (Spring  5th  grade) 


School  sample  size 

12 

12 

12 

12 

12 

School  year  dates 
First  day  of  school 
Start  of  implementation 
Last  day  of  school 

8/30/04 

9/6/04 

t 

t 

t 

6/1 6/05 

8/29/05 
First  day 
t 

t 

t 

6/1 5/06 

8/28/06 
First  day 
6/14/07 

Data  collection 
Start 
End 

10/11/04 
1 0/29/04 

4/25/05 

5/13/05 

1 0/24/05 
11/11/05 

4/24/06 
5/1 2/06 

4/23/07 
5/1 0/07 

Calendar  weeks  from 
program  implementation 
to  start  of  fall  2004 
data  collection 

5 

t 

t 

t 

t 

Calendar  weeks  from 
start  of  school  to  start  of 
fall  2004  data  collection 

6 

t 

8 

t 

t 

Calendar  weeks  from 
end  of  fall  data  collection 
to  start  of  spring  data 
collection 

t 

25 

t 

23 

t 

Calendar  weeks  from 
program  implementation 
to  start  of  spring  data 
collection 

t 

33 

t 

34 

34 

t Not  applicable. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 


Consent  Rates,  Completion  Rates,  and  Percentage  of  Sample  With  Data 

The  actual  number  of  student,  primary  caregiver,  and  teacher  reports  available  for  analysis  was  smaller  than 
the  number  in  the  sample  because  consent  and  completion  rates  were  less  than  100  percent.  Primary 
caregivers  had  to  provide  consent  before  children  could  complete  the  Child  Report,  before  their  child’s 
teacher  could  complete  the  Teacher  Report  on  Student,  and  before  they  themselves  completed  the  Primary 
Caregiver  Report.  There  were  statistically'  significant  differences  in  consent  for  child  and  primary  caregiver  in 
Year  1;  treatment  caregivers  had  higher  consent  rates  than  control  caregivers.  Teachers  also  had  to  provide 
consent  before  completing  the  Teacher  Report  on  Classroom  and  School.  There  were  no  statistically 
significant  differences  between  treatment  and  control  groups  in  consent  rates  for  teachers. 

Of  those  with  consent,  not  all  completed  their  respective  reports.  Table  8.4  shows  the  consent  rates, 
completion  rates,  and  percentages  of  sample  with  data  for  each  of  the  four  reports  over  the  3 years.  For  the 
Child  Report  and  two  teacher  reports,  completion  rates  ranged  from  87  percent  to  100  percent,  with  no 
statistically  significant  difference  in  completion  rates  between  treatment  and  control  groups.  For  the  Primary 
Caregiver  Report,  the  completion  rates  dropped  over  time  from  96  percent  to  83  percent.  There  was  one 
statistically  significant  difference  for  the  Primary  Caregiver  Report  completion  rate;  in  the  spring  of  Year  1, 
fewer  treatment  group  caregivers  completed  reports  than  did  control  group  caregivers. 
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The  percentages  of  the  sample  with  Child  Report  data  ranged  from  58  percent  to  72  percent  over  the  3 years, 
with  one  statistically  significant  difference  in  the  fall  of  Year  1.  The  percentages  of  students  with  information 
from  the  Teacher  Report  on  Student  ranged  from  60  percent  to  72  percent,  again  with  one  statistically 
significant  difference  in  the  fall  of  Year  1.  The  percentages  of  students  with  data  from  the  Primary  Caregiver 
Report  ranged  from  55  percent  to  66  percent,  also  with  one  statistically  significant  difference  in  the  fall  of 
Year  1.  The  percentages  of  teachers  with  data  from  the  Teacher  Report  on  Classroom  and  School  ranged 
from  87  percent  to  97  percent,  with  no  statistically  significant  differences  between  treatment  and  control 
teachers. 
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Table  8.4.  Consent  rates,  completion  rates,  and  percentage  of  sample  with  data  from  each  report — SS 


Year  1 Year  1 Year  2 Year  3 

(Fall  3rd  grade)  (Spring  3rd  grade)  (Spring  4th  grade)  (Spring  5th  grade) 

Treat-  Treat-  Treat-  Treat- 


Report 

Total 

ment 

Control 

Total 

ment 

Control 

Total 

ment 

Control 

Total 

ment 

Control 

Student  sample  size 

944 

524 

420 

951 

528 

423 

959 

533 

426 

989 

533 

456 

Child  Report  (percent) 

Primary  caregiver  consent  rate 

65.0 

68.9** 

60.2 

65.7 

68.6* 

62.2 

71.1 

72.4 

69.5 

68.4 

70.4 

66.0 

Student  completion  rate 

95.9 

95.6 

96.4 

99.8 

99.7 

100.0 

99.9 

100.0 

99.7 

99.6 

100.0 

99.0 

Students  with  data1 

62.3 

65.7 

58.0 

65.6 

68.4 

62.2 

71.0 

72.4 

69.2 

68.0 

70.4 

65.4 

Primary  Caregiver  Report  (percent) 

Primary  caregiver  consent  rate 

65.0 

68.9** 

60.2 

65.7 

68.8* 

61.9 

70.6 

71.9 

69.0 

68.1 

70.0 

66.0 

Primary  caregiver  completion  rate 

95.8 

95.6 

96.0 

87.2 

84.3* 

91.2 

87.6 

88.0 

87.1 

83.2 

83.4 

83.1 

Primary  caregivers  with  data1 

62.3 

65.7 

57.8 

57.3 

58.0 

56.5 

61.8 

63.2 

60.1 

56.7 

58.3 

54.8 

Teacher  Report  on  Student  (percent) 

Primary  caregiver  consent  rate2 

65.0 

68.9** 

60.2 

65.7 

68.6* 

62.2 

71.1 

72.4 

69.5 

68.4 

70.4 

66.0 

Teacher  completion  rate 

99.7 

99.4 

100.0 

99.8 

99.7 

100.0 

99.6 

99.5 

99.7 

99.7 

100.0 

99.3 

Students  with  data1 

64.8 

68.5 

60.2 

65.6 

68.4 

62.2 

70.8 

72.0 

69.2 

68.1 

70.4 

65.6 

Teacher  Report  on  Classroom  and  School 
(3rd-  to  5th-grade  teachers)  (percent) 

Teacher  consent  rate 

100.0 

100.0 

100.0 

100.0 

100.0 

100.0 

100.0 

100.0 

100.0 

100.0 

100.0 

100.0 

Teacher  completion  rate 

92.2 

90.3 

94.4 

93.1 

93.5 

92.6 

96.6 

96.8 

96.2 

89.0 

90.8 

86.8 

Teachers  with  data1 

92.2 

90.3 

94.4 

93.1 

93.5 

92.6 

96.6 

96.8 

96.2 

89.0 

90.8 

86.8 

* Treatment  group  significantly  different  from  control  group  at  the  .05  level. 

**  Treatment  group  significantly  different  from  control  group  at  the  .01  level. 

1 Calculated  as  consent  rate  x completion  rate. 

2The  primary  caregiver  consent  rates  for  the  Child  Report  and  the  Teacher  Report  on  Student  are  identical,  as  the  primary  caregiver  gave  consent  to  both  together. 
SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Responses  from  students  in  the  original  cohort  (stayers)  and  new  entrants  in  the  SS  sample  were  examined  to 
investigate  possible  differences  between  the  two  groups  in  consent  rates,  completion  rates,  and  the 
percentages  of  sample  with  data  that  might  affect  outcome  data  (table  8.5).  In  Years  1 and  2,  the  stayers  had 
significantly  higher  consent  rates  than  the  new  entrants  (by  18  to  24  percentage  points),  and  in  Year  3,  the 
opposite  trend  was  found,  with  new  entrants  having  significantly  higher  consent  rates  than  stayers  (by  18 
percentage  points).  These  differences  in  consent  rates  resulted  in  a similar  pattern  of  differences  for 
percentages  of  sample  with  data,  with  more  data  for  the  stayers  in  Years  1 and  2 on  all  three  reports  (by  17  to 
22  percentage  points),  and  more  data  for  the  new  entrants  in  Year  3 on  all  three  reports  (by  15  to  16 
percentage  points). 
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Table  8.5.  Consent  rates,  completion  rates,  and  percentage  of  sample  with  data:  Stayers  versus  new  entrants — SS 


Year  1 Year  2 Year  3 

(Spring  3rd  grade)  (Spring  4th  grade)  (Spring  5th  grade) 


Report 

Total 

Stayers 

New 

entrants 

Total 

Stayers 

New 

entrants 

Total 

Stayers 

New 

entrants 

Student  sample  size 

951 

918 

33 

959 

826 

133 

989 

855 

134 

Child  Report  (percent) 

Primary  caregiver  consent  rate 

65.7 

66.4* 

45.5 

71.7 

73.7*** 

54.9 

68.4 

66.0*** 

83.6 

Student  completion  rate 

99.8 

99.8 

100.0 

99.9 

99.8 

100.0 

99.6 

99.8* 

98.2 

Students  with  data1 

65.6 

66.3* 

45.5 

71.0 

73.6*** 

54.9 

68.0 

65.8*** 

82.1 

Primary  Caregiver  Report  (percent) 

Primary  caregiver  consent  rate 

65.7 

66.6** 

42.4 

70.6 

73.1*** 

54.9 

68.1 

65.7*** 

83.6 

Primary  caregiver  completion  rate 

87.2 

87.2 

85.7 

87.6 

87.7 

86.3 

83.4 

83.5 

83.0 

Primary  caregivers  with  data1 

57.3 

58.1* 

36.4 

61.8 

64.2*** 

47.4 

56.8 

54.9** 

69.4 

Teacher  Report  on  Student  (percent) 

Primary  caregiver  consent  rate2 

65.7 

66.4* 

45.5 

71.7 

73.7*** 

54.9 

68.4 

66.0*** 

83.6 

Teacher  completion  rate 

99.8 

99.8 

100.0 

99.6 

99.7 

98.6 

99.7 

100.0** 

98.2 

Students  with  data1 

65.6 

66.3* 

45.5 

70.8 

73.5*** 

54.1 

68.1 

66.0*** 

82.1 

* Stayers  significantly  different  from  new  entrants  at  the  .05  level. 

**  Stayers  significantly  different  from  new  entrants  at  the  .01  level. 

***  Stayers  significantly  different  from  new  entrants  at  the  .001  level. 

1 Calculated  as  consent  rate  x completion  rate. 

2The  primary  caregiver  consent  rates  for  the  Child  Report  and  the  Teacher  Report  on  Student  are  identical,  as  the  primary  caregiver  gave  consent  to  both  together. 
SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 


Chapter  8.  Second  Step 


Chapter  8.  Second  Step 


Fidelity  of  Implementation 

In  Years  2 and  3,  the  six  SS  treatment  group  schools  were  individually  rated  for  the  quantity  and  quality  of  the 
program’s  implementation  by  two  raters  from  the  research  team  (in  Year  1,  only  one  rater  was  used).  The 
global  measure  of  fidelity  for  the  multisite  study  was  used;  inter-rater  reliability  was  measured  using 
Cronbach’s  alpha  (Year  2 = 1.00  and  Year  3 = 0.99).  The  ratings  were  combined  into  a single  consensus 
rating  and  used  to  identify  schools  with  high  implementation  fidelity.  In  Year  2,  five  treatment  schools  were 
identified  as  having  high  fidelity,  and  in  Year  3 four  schools  were.  Cohen’s  kappa  was  used  as  the  measure  of 
agreement  when  identifying  schools  as  high  fidelity,  and  it  equaled  1.00  in  both  years.  In  Year  1,  four 
treatment  schools  were  identified  as  having  high  fidelity;  however,  only  one  rater  was  used,  and  therefore 
inter-rater  reliability  could  not  be  determined. 

Initial  Characteristics 

This  section  examines  the  initial  characteristics  of  the  students,  teachers,  and  schools  participating  in  the 
evaluation  of  the  SS  program.  These  characteristics  were  collected  from  students  who  were  enrolled  in  the 
third  grade  at  the  study  schools  in  fall  2004,  as  well  as  from  their  primary  caregivers  and  third-grade  teachers. 
In  addition,  third-,  fourth-,  and  fifth-grade  teachers  and  principals  in  the  study  schools  provided  information 
about  activities  related  to  social  and  character  development  in  these  schools.  Documenting  the  characteristics 
of  students,  teachers,  and  schools  and  initial  measures  of  key  outcomes  at  a point  before  the  interventions 
had  been  operating  for  an  extended  period  helped  to  determine  whether  the  random  assignment  of  schools  to 
treatment  and  control  status  produced  treatment  and  control  groups  with  similar  distributions  of  observed 
characteristics.  As  noted  in  the  following  discussion,  there  were  13  significant  differences  in  the  observed 
characteristics,  12  of  which  (out  of  62  comparisons,  with  3 expected  to  be  significant  by  chance)  were  related 
to  differences  between  the  treatment  and  control  students,  teachers,  and  schools  in  the  use  of  SACD 
activities  in  the  classroom  and  school. 

Characteristics  of  Children,  Their  Families,  and  Communities 

The  mean  age  of  students  in  the  overall  SS  sample  was  8.0  years.  The  sample  contained  equal  percentages  of 
girls  (50%)  and  boys  (50%).  The  sample  had  some  ethnic  diversity,  with  White  non-Hispanic  students  making 
up  83  percent  of  the  sample,  Black  non-Hispanic  students  making  up  7 percent  of  the  sample,  and  Hispanic 
students  making  up  5 percent  of  the  sample. 

The  sample  was  also  somewhat  diverse  in  its  levels  of  family  income,  education  levels  of  primary  caregivers  of 
the  children  in  the  sample,  and  family  situation.  For  the  total  sample,  5 percent  of  children  lived  in  a 
household  where  the  income  was  135  percent  of  the  federal  poverty  level  or  lower,  which  is  the  income 
threshold  for  eligibility  for  free  school  meals.  About  4 percent  of  primary  caregivers  had  not  completed  high 
school.  There  was  a statistically  significant  difference  between  the  groups,  with  more  control  group  caregivers 
not  having  completed  high  school  than  treatment  group  caregivers  (5%  versus  4%).  The  majority  of  the 
children  (80%)  lived  with  both  their  mother  and  their  father. 

The  mean  values  of  the  outcomes  for  children’s  behavior  and  attitudes  as  reported  by  the  primary  caregiver, 
child,  and  teacher  at  initial  data  collection  in  fall  2004  are  shown  in  table  8.7.  There  were  no  significant 
differences  between  the  treatment  and  control  groups. 
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Table  8.6.  Initial  characteristics  of  children,  their  families,  and  communities — SS 


Characteristic 

Total 

Treatment 

Control 

Student  sample  size 

588 

345 

243 

Student  demographics 

Gender  (percent) 

Male 

50.4 

51.6 

49.3 

Female 

49.6 

48.4 

50.7 

Race/ethnicity  (percent) 

White  (non-Hispanic) 

82.9 

85.8 

80.0 

Black  (non-Hispanic) 

6.6 

5.3 

7.8 

Hispanic 

4.7 

2.9 

6.5 

Other 

5.8 

6.0 

5.6 

Age  (in  years)  (mean) 

8.0 

8.0 

8.0 

Primary  caregiver  and  family  characteristics 

Primary  caregiver’s  age  (in  years)  (mean) 

38.6 

38.3 

38.9 

Primary  caregiver’s  race/ethnicity  (percent) 

White  (non-Hispanic) 

84.7 

88.4 

80.9 

Black  (non-Hispanic) 

7.0 

5.6 

8.3 

Hispanic 

4.2 

2.0 

6.4 

Other 

4.2 

3.9 

4.4 

Primary  caregiver’s  education  (percent) 

Did  not  complete  high  school 

4.1 

3.5* 

4.6 

Completed  high  school  or  equivalent 

13.5 

18.2 

8.8 

Some  college 

36.1 

34.5 

37.6 

Bachelor’s  or  higher  degree 

46.4 

43.8 

49.0 

Primary  caregiver’s  employment  (percent) 

Full-time 

49.2 

50.3 

48.0 

Other 

50.8 

49.7 

52.0 

Primary  caregiver’s  marital  status  (percent) 

Married 

79.3 

79.4 

79.3 

Other 

20.7 

20.6 

20.7 

Students  who  live  in  one  household  (percent) 

96.1 

95.0 

97.1 

Number  of  individuals  in  household  (mean) 

4.7 

4.6 

4.8 

Primary  caregiver’s  relationship  to  child  (percent) 

Mother  (stepmother) 

90.1 

90.1 

90.1 

Father  (stepfather) 

7.6 

6.9 

8.3 

Other  relative/nonrelative 

2.3 

3.0 

1.5 

See  notes  at  end  of  table. 
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Table  8.6.  Initial  characteristics  of  children,  their  families,  and  communities — SS — Continued 


Characteristic 

Total 

Treatment 

Control 

Student  lives  with  (percent) 

Mother  (stepmother)  and  father  (stepfather) 

80.3 

79.3 

81.2 

Mother  (stepmother)  only;  father  (stepfather)  not  present 

15.3 

15.9 

14.8 

Father  (stepfather)  only;  mother  (stepmother)  not  present 

2.8 

2.7 

3.0 

Other  relative/nonrelative,  parents  not  present 

1.6 

t 

t 

Highest  education  of  anyone  in  household  (percent) 

Did  not  complete  high  school 

1.4 

1.5 

1.2 

Completed  high  school  or  equivalent 

10.0 

13.0 

7.0 

Some  college 

31.3 

30.5 

32.1 

Bachelor’s  or  higher  degree 

57.3 

54.9 

59.7 

Total  household  income  (percent) 

Less  than  $20,000 

2.2 

1.7 

2.6 

$20,000  to  $39,999 

10.1 

10.3 

10.0 

$40,000  to  $59,999 

11.2 

12.3 

10.2 

$60,000  or  more 

76.5 

75.7 

77.2 

Income-to-poverty-threshold  ratio — Below  135  percent  (percent) 

5.1 

4.9 

5.3 

Income-to-poverty-threshold  ratio — 135  to  185  percent  (percent) 

7.2 

7.5 

6.8 

Income-to-poverty-threshold  ratio — Above  185  percent  (percent) 

87.7 

87.6 

87.8 

Alabama  Parenting  Questionnaire — Poor  Monitoring  and 

Supervision  Subscale  (mean) 

1.1 

1.1 

1.1 

Alabama  Parenting  Questionnaire — Positive  Parenting 

Subscale  (mean) 

3.5 

3.5 

3.5 

Confusion,  Hubbub,  and  Order  Scale  (mean) 

2.2 

2.3 

2.2 

Community  characteristics  (mean) 

Community  Risks  Scale 

1.1 

1.1 

1.1 

Community  Resources  Scale 

3.1 

3.0 

3.1 

Child-Centered  Social  Control  Scale 

3.5 

3.5 

3.5 

$ Reporting  standards  not  met.  Values  suppressed  to  protect  confidentiality. 

* Treatment  group  significantly  different  from  control  group  at  the  .05  level. 

NOTE:  Weights,  which  assign  equal  weight  to  each  school  within  the  program,  were  used  in  producing  the  treatment,  control,  and 
overall  means.  Statistical  tests  were  conducted  using  regressions  that  included  program  indicators  to  account  for  the  sample  design 
and  adjusted  for  clustering  at  the  school  level. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Table  8.7.  Mean  scores  and  standard  deviations  for  initial  outcome  measures  of  sample — SS 


Outcome  measure-Report 

Range 

Total 

Mean 

SD 

Treatment 
Mean  SD 

Control 
Mean  SD 

Social  and  Emotional  Competence  Domain 

Self-Efficacy  for  Peer  Interaction-CR 

1-4 

3.0 

0.6 

3.0 

0.6 

3.0 

0.6 

Normative  Beliefs  About  Aggression-CR 

1-4 

1.2 

0.4 

1.2 

0.4 

1.2 

0.4 

Empathy-CR 

1-3 

2.4 

0.4 

2.5 

0.3 

2.4 

0.4 

Behavior  Domain 

Altruistic  Behavior-Child 

0-3 

1.2 

0.8 

1.2 

0.8 

1.2 

0.8 

Altruistic  Behavior-TRS 

1-4 

1.4 

0.5 

1.5 

0.6 

1.4 

0.4 

Altruistic  Behavior-PCR 

1-4 

2.1 

0.7 

2.2 

0.7 

2.1 

0.7 

Positive  Social  Behavior-TRS 

1-4 

3.4 

0.6 

3.4 

0.6 

3.4 

0.6 

Positive  Social  Behavior-PCR 

1-4 

3.1 

0.5 

3.2 

0.5 

3.1 

0.5 

Problem  Behavior-CR 

0-3 

0.1 

0.2 

0.1 

0.2 

0.1 

0.2 

Problem  Behavior-TRS 

1-4 

1.2 

0.3 

1.2 

0.3 

1.2 

0.3 

Problem  Behavior-PCR 

1-4 

1.5 

0.3 

1.5 

0.3 

1.5 

0.3 

ADHD-Related  Behavior-TRS 

1-4 

1.5 

0.6 

1.5 

0.6 

1.5 

0.6 

Academics  Domain 

Academic  Competence  and  Motivation-TRS 

1-5 

3.3 

0.8 

3.3 

0.8 

3.4 

0.8 

Engagement  with  Learning-CR 

1-4 

3.8 

0.4 

3.8 

0.4 

3.8 

0.4 

Perceptions  of  School  Climate  Domain 

Positive  School  Orientation-CR 

1-4 

3.4 

0.5 

3.4 

0.5 

3.3 

0.5 

Negative  School  Orientation-CR 

1-4 

1.5 

0.5 

1.5 

0.4 

1.5 

0.5 

Student  Afraid  at  School-CR 

1-4 

2.1 

0.9 

2.1 

0.9 

2.1 

0.9 

Victimization  at  School-CR 

0-3 

0.6 

0.7 

0.6 

0.7 

0.6 

0.7 

Student  sample  size — PCR 

588 

345 

243 

Student  sample  size — CR 

589 

345 

244 

Student  sample  size — TRS 

612 

359 

253 

NOTE:  Abbreviations  are 
CR:  Child  Report 
PCR:  Primary  Caregiver  Report 
TRS:  Teacher  Report  on  Student 
ADHD:  Attention  deficit  hyperactivity  disorder 
SD:  Standard  deviation 

No  statistically  significant  differences  were  found  between  values  for  treatment  and  control  groups.  Weights,  which  assign  equal 
weight  to  each  school  within  the  program,  were  used  in  producing  the  treatment,  control,  and  overall  means.  Statistical  tests  were 
conducted  using  regressions  that  included  program  indicators  to  account  for  the  sample  design  and  adjusted  for  clustering  at  the 
school  level.  Sample  size  may  differ  for  some  outcomes  due  to  nonresponse. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Characteristics  of  Teachers  and  Schools 

Table  8.8  describes  the  third-,  fourth-,  and  fifth-grade  teachers  at  the  study  schools.  The  majority  were  White 
non-Hispanic  (92%)  and  female  (96%).  They  had  an  average  of  12.3  years  of  total  teaching  experience,  and 
more  than  half  (59%)  held  an  advanced  or  specialist  degree.  There  were  no  significant  differences  between 
the  treatment  and  control  groups  in  these  characteristics. 

Data  regarding  school  characteristics  were  drawn  from  the  Common  Core  of  Data  in  order  to  compare 
treatment  and  control  schools.  There  were  no  significant  differences  between  the  two  groups  of  schools  in 
terms  of  student  composition  (race/ ethnicity  and  school  lunch  eligibility),  number  of  students  enrolled, 
number  of  full-time  teachers,  Title  I status,  or  number  of  years  the  principal  had  been  at  the  school  (see  table 
8.9).  In  addition,  there  were  no  significant  differences  between  treatment  and  control  schools  in  terms  of 
location  (urban,  suburban,  or  rural)  or  lowest  and  highest  grade  offered  (these  data  are  not  shown  in  a table). 

Table  8.8.  Initial  characteristics  of  teachers  in  sample — SS 


Characteristic 

Total 

Treatment 

Control 

Teacher  sample  size 

107 

56 

51 

Gender  (percent) 

Male 

4.0 

4 

4 

Female 

96.0 

4 

4 

Race/ethnicity  (percent) 

White  (non-Hispanic) 

92.1 

93.9 

90.3 

Other 

7.9 

6.1 

9.7 

Number  of  years  teaching  experience  (mean) 

12.3 

12.8 

11.8 

Number  of  years  teaching  experience  this  school  (mean) 

7.5 

8.2 

6.7 

Type  of  teaching  certificate  (percent) 

Regular  state  certificate  or  advanced  professional  certificate 

97.5 

95.1 

100.0 

Other 

2.5 

4.9 

0.0 

Education  (percent) 

Bachelor’s  degree 

41.0 

36.6 

45.4 

Advanced  degree/other 

59.0 

63.4 

54.6 

$ Reporting  standards  not  met.  Values  suppressed  to  protect  confidentiality. 

NOTE:  No  statistically  significant  differences  were  found  between  values  for  treatment  and  control  groups.  Weights,  which  assign 
equal  weight  to  each  school  within  the  program,  were  used  in  producing  the  treatment,  control,  and  overall  means.  Statistical  tests 
were  conducted  using  regressions  that  included  program  indicators  to  account  for  the  sample  design  and  adjusted  for  clustering  at 
the  school  level.  Sample  size  may  differ  for  some  outcomes  due  to  nonresponse. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Table  8.9.  Initial  characteristics  of  schools  in  sample — SS 


Characteristic 

Total 

Treatment 

Control 

School  sample  size 

12 

6 

6 

Student  race/ethnicity  (percent) 

White  (non-Hispanic) 

83.7 

88.2 

79.2 

Black  (non-Hispanic) 

10.4 

7.1 

13.6 

Hispanic 

2.2 

1.2 

3.3 

Other 

3.7 

3.5 

3.8 

Students  eligible  for  free  or  reduced-price  lunch  (percent) 

12.2 

11.7 

12.6 

Number  of  students  enrolled  (mean) 

453.8 

491.8 

415.7 

Number  of  full-time  teachers  (mean) 

26.1 

27.1 

25.1 

Title  1 status  (percent) 

Title  1 eligible  school 

16.7 

16.7 

16.7 

Schoolwide  Title  1 

0.0 

0.0 

0.0 

Number  of  years  principal  has  been  at  this  school  (mean) 

4.2 

4.0 

4.4 

NOTE:  No  statistically  significant  differences  were  found  between  values  for  treatment  and  control  groups.  Weights,  which  assign 
equal  weight  to  each  school  within  the  program,  were  used  in  producing  the  treatment,  control,  and  overall  means.  Statistical  tests 
were  conducted  using  regressions  that  included  program  indicators  to  account  for  the  sample  design  and  adjusted  for  clustering  at 
the  school  level.  Sample  size  may  differ  for  some  outcomes  due  to  nonresponse. 

SOURCE:  NCES  Common  Core  of  Data  (2003-04),  the  Social  and  Character  Development  (SACD)  Research  Program. 

In  the  Teacher  Report  on  Classroom  and  School,  teachers  reported  on  nine  dimensions  of  school 
environment  (these  data  are  not  shown  in  a table):  feelings  of  safety,  adequacy  of  resources,  student  support, 
freedom  to  teach  as  desired,  affiliation  with  and  ties  to  colleagues,  innovation  regarding  new  approaches  to 
teaching,  professional  interest,  participatory  decisionmaking,  and  work  pressure.  There  were  no  statistically 
significant  differences  between  treatment  and  control  schools  in  these  reports. 

The  Level  of  SACD  in  the  Schools  Near  the  Beginning  of  the  Study 

During  the  initial  data  collection,  principals  and  teachers  reported  on  the  SACD  activities  used  in  the  schools 
and  classrooms,  the  availability  of  SACD  materials,  and  the  professional  development  provided  on  SACD. 
Table  8.10  shows  that  the  majority  of  the  school  principals  reported  activities  to  promote  six  social  and 
character  development  goals:  violence  prevention  and  peace  promotion  (92%),  social  and  emotional 
development  (100%),  character  education  (100%),  tolerance  and  diversity  (92%),  risk  prevention  and  health 
promotion  (75%),  and  civic  responsibility  and  community  service  (83%).  In  addition,  all  of  the  principals 
reported  activities  directed  toward  behavior  management.  There  were  no  statistically  significant  differences 
between  the  treatment  and  the  control  group  schools  in  the  percentages  reported  by  principals,  although  this 
may  be  due  to  the  small  principal  sample  size.  Teachers  reported  that  use  of  these  activities  in  their 
classrooms  ranged  from  47  percent  to  92  percent,  with  three  significant  differences  (out  of  eight 
comparisons)  between  treatment  and  control  teachers:  larger  percentages  of  treatment  teachers  than  control 
teachers  reported  the  use  of  social  and  emotional  development  (89%  versus  61%)  and  character  education 
(99%  versus  85%)  activities  in  their  classrooms,  while  more  control  than  treatment  teachers  reported  the  use 
of  civic  responsibility  and  community  service  activities  (69%  versus  48%)  in  their  classrooms.  With  respect  to 
the  use  of  schoolwide  activities,  62  percent  to  96  percent  of  teachers  reported  that  their  schools  used  such 
activities.  There  were  no  significant  differences  between  treatment  and  control  groups. 
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Table  8.10.  Principal  and  teacher  initial  reports  on  use  of  SACD  programs  or  activities  in 
sample — SS 

SACD  program  or  activity Total Treatment Control 


Principal  sample  size 

12 

6 

6 

Teacher  sample  size 

107 

56 

51 

Principals  reporting  that  school  had  programs  or  activities 
to  promote  the  following  SACD  goals  (percent) 

Violence  prevention  and  peace  promotion 

91.7 

100.0 

83.3 

Social  and  emotional  development 

100.0 

100.0 

100.0 

Character  education 

100.0 

100.0 

100.0 

Tolerance  and  diversity 

91.7 

100.0 

83.3 

Risk  prevention  and  health  promotion 

75.0 

83.3 

66.7 

Civic  responsibility  and  community  service 

83.3 

100.0 

66.7 

Behavior  management 

100.0 

100.0 

100.0 

None  of  the  above 

0.0 

0.0 

0.0 

Teachers  reporting  on  using  programs  or  activities  in  their  class 
to  promote  the  following  SACD  goals  (percent) 

Violence  prevention  and  peace  promotion 

57.6 

62.2 

53.1 

Social  and  emotional  development 

74.9 

89.0** 

60.9 

Character  education 

91.9 

98.6* 

85.1 

Tolerance  and  diversity 

58.8 

60.5 

57.0 

Risk  prevention  and  health  promotion 

46.9 

46.6 

47.2 

Civic  responsibility  and  community  service 

58.7 

48.1* 

69.3 

Behavior  management 

89.4 

91.5 

87.3 

None  of  the  above 

t 

0.0 

t 

Teachers  reporting  schoolwide  use  of  the  following 
activities  to  promote  SACD  (percent) 

Morning  announcements  or  videos 

96.2 

94.8 

97.6 

School  assemblies 

65.1 

56.2 

73.9 

School  newspapers  or  bulletins 

84.1 

83.0 

85.2 

Special  school  days 

61.8 

62.4 

61.1 

Special  events 

68.1 

65.4 

70.8 

Other  activities 

10.0 

13.7 

6.4 

$ Reporting  standards  not  met.  Values  suppressed  to  protect  confidentiality. 


* Treatment  group  significantly  different  from  control  group  at  the  .05  level. 

**  Treatment  group  significantly  different  from  control  group  at  the  .01  level. 

NOTE:  Weights,  which  assign  equal  weight  to  each  school  within  each  of  the  programs  and  to  each  program  across  programs,  were 
used  in  producing  the  treatment,  control,  and  overall  means.  Statistical  tests  were  conducted  using  regressions  that  included 
program  indicators  to  account  for  the  sample  design  and  adjusted  for  clustering  at  the  school  level.  Sample  size  may  differ  for  some 
outcomes  due  to  nonresponse. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Teachers  reported  using  a broad  range  of  teaching  materials  to  support  SACD  activities  (table  8.1 1),  including 
teacher  guides  (85%),  student  materials  (45%),  instructional  aids  (35%),  giveaways  (54%),  and  children’s 
literature  (38%).  There  was  one  statistically  significant  difference  (out  of  seven  comparisons)  in  the  use  of 
instructional  aids,  with  more  treatment  teachers  than  control  teachers  reporting  their  use  (52%  versus  18%). 

Teachers  also  reported  using  a wide  variety  of  teaching  strategies  (table  8.11).  All  teachers  reported  using  any 
of  the  20  strategies  asked  about,  and  teachers  used  an  average  of  12  of  the  strategies.  There  were  4 significant 
differences  (out  of  20  comparisons)  between  treatment  and  control  teachers  in  the  use  of  these  specific 
strategies.  More  treatment  teachers  than  control  teachers  reported  using  role-playing  (98%  versus  44%),  direct 
instruction  of  SACD  (98%  versus  73%),  skill  training  (68%  versus  34%),  and  guided  visualization  (65% 
versus  29%). 
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Table  8.11.  Teacher  initial  reports  on  use  of  SACD  materials  and  classroom  strategies  in  the 
sample — SS 


SACD  material  and  classroom  strategy 

Total 

Treatment 

Control 

Teacher  sample  size 

107 

56 

51 

Teachers  using  the  following  materials  in  conjunction  with 
social  and  character  development  activities  (percent) 

Teacher  guides  (manuals,  curricula) 

84.9 

93.5 

76.3 

Student  materials  (workbooks,  worksheets) 

45.2 

47.8 

42.6 

Instructional  aids  (games,  software,  videos) 

34.9 

52.1** 

17.8 

Giveaways  (bookmarks,  stickers) 

54.0 

49.8 

58.3 

Children’s  literature 

37.5 

43.7 

31.4 

Other  types  of  materials 

13.6 

11.6 

15.5 

Do  not  use  any  of  the  materials  listed  above 

4.8 

t 

t 

Teachers  using  any  of  the  strategies  listed  below  to  promote 

social  and  character  development  in  the  classroom  (percent) 

100.0 

100.0 

100.0 

Number  of  strategies  (listed  below)  used  by  teachers  to  promote 

social  and  character  development  in  the  classroom  (mean) 

12.0 

12.6 

11.3 

Teachers  using  each  of  the  following  strategies  to  promote 
social  and  character  development  (percent) 

Role-playing 

71.0 

98.3** 

43.6 

Cooperative  learning 

95.6 

98.3 

92.9 

Peer  group  discussions 

85.7 

90.4 

81.0 

Direct  instruction  of  social  and  character  development 

85.5 

97.9* 

73.0 

Skill  training 

51.1 

68.4* 

33.8 

Incorporating  social  and  character  development  into 

academic  curriculum 

80.1 

80.4 

79.8 

Parent  training 

10.3 

11.0 

9.7 

Parent/community  involvement  in  program  development 

or  delivery 

36.0 

37.8 

34.2 

Mentoring 

43.3 

37.0 

49.7 

Good  behavior  notes  sent  home  daily  or  weekly 

77.2 

75.1 

79.3 

Presenting  role  models 

69.3 

74.6 

64.0 

Targeted  story  reading  or  writing  on  SACD  themes 

67.1 

73.8 

60.3 

Peer  mediation 

30.1 

26.2 

34.0 

Honor  roll  for  positive  behavior 

42.7 

36.6 

48.9 

Pledges  or  recitations  on  social  and  character 

development  themes 

34.6 

35.0 

34.1 

Guided  visualization 

47.0 

65.2* 

28.9 

Student-led/student-assisted  instruction 

43.4 

38.6 

48.2 

Journaling 

66.4 

68.7 

64.1 

Time  out  for  negative  behavior 

80.0 

80.0 

80.1 

Daily  or  weekly  rewards  for  positive  behavior 

89.5 

87.6 

91.5 

t Reporting  standards  not  met.  Values  suppressed  to  protect  confidentiality. 

* Treatment  group  significantly  different  from  control  group  at  the  .05  level. 

**  Treatment  group  significantly  different  from  control  group  at  the  .01  level. 

NOTE:  Weights,  which  assign  equal  weight  to  each  school  within  each  of  the  programs  and  to  each  program  across  programs,  were 
used  in  producing  the  treatment,  control,  and  overall  means.  Statistical  tests  were  conducted  using  regressions  that  included 
program  indicators  to  account  for  the  sample  design  and  adjusted  for  clustering  at  the  school  level.  Sample  size  may  differ  for  some 
outcomes  due  to  nonresponse. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Principals  and  teachers  reported  on  participation  in  and  amount  of  SACD  training  and  staff  development 
provided  over  the  previous  12  months  (table  8.12).  Principals  reported  higher  participation  rates  (100% 
versus  81%)  and  more  hours  of  training  (9.7  versus  7.6)  than  did  teachers.  There  was  a significant  difference 
between  treatment  and  control  teachers  on  the  percentages  reporting  participation  in  SACD  training;  this 
favored  the  treatment  group  (96%  versus  67%).  The  treatment  teachers  also  reported  significandy  more 
training  than  did  the  control  teachers  on  three  out  of  seven  specific  SACD  areas:  violence  prevention  and 
peace  promotion  (24%  versus  9%),  social  and  emotional  development  (43%  versus  13%),  and  character 
education  (90%  versus  50%). 


Table  8.12.  Principal  and  teacher  initial  reports  on  SACD  professional  development  in  sample — 
SS 


SACD  professional  development 

Total 

Treatment 

Control 

Principal  sample  size 

12 

6 

6 

Teacher  sample  size 

107 

56 

51 

Principals  reporting  that  staff  participated  in  social  and  character 
development  training  within  the  past  year  (percent) 

100.0 

100.0 

100.0 

Teachers  reporting  participation  in  social  and  character 
development  training  within  the  past  12  months  (percent) 

81.4 

95.6" 

67.2 

Number  of  hours  of  social  and  character  development  training 
principals  report  were  provided  to  each  staff  person  last  year  (mean) 

9.7 

8.2 

11.6 

Number  of  hours  of  social  and  character  development  training 
teachers  report  receiving  during  the  past  12  months  (mean) 

7.6 

9.3 

5.8 

Teachers  reporting  receiving  training  in  the  past  12  months 
in  the  following  areas  (percent) 

Violence  prevention  and  peace  promotion 

16.5 

24.2* 

8.8 

Social  and  emotional  development 

27.7 

42.5** 

12.9 

Character  education 

69.9 

89.8** 

50.0 

Tolerance  and  diversity 

11.6 

7.7 

15.5 

Risk  prevention  and  health  promotion 

14.7 

13.1 

16.4 

Civic  responsibility  and  community  service 

3.1 

t 

t 

Behavior  management 

25.4 

19.0 

31.8 

4:  Reporting  standards  not  met.  Values  suppressed  to  protect  confidentiality. 


* Treatment  group  significantly  different  from  control  group  at  the  .05  level. 

**  Treatment  group  significantly  different  from  control  group  at  the  .01  level. 

NOTE:  Weights,  which  assign  equal  weight  to  each  school  within  each  of  the  programs  and  to  each  program  across  programs,  were 
used  in  producing  the  treatment,  control,  and  overall  means.  Statistical  tests  were  conducted  using  regressions  that  included 
program  indicators  to  account  for  the  sample  design  and  adjusted  for  clustering  at  the  school  level.  Sample  size  may  differ  for  some 
outcomes  due  to  nonresponse. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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The  data  on  the  initial  level  of  SACD  activity  emphasize  that  the  control  condition  was  a “standard  practice” 
control.  Standard  practice  at  the  control  schools  included  using  SACD  activities,  materials,  and  practices, 
along  with  professional  development,  at  rates  and  in  types  and  amounts  similar  to  the  treatment  schools.  For 
example,  the  percentages  of  teachers  who  reported  using  programs  or  activities  to  promote  specific  SACD 
goals  ranged  from  47  percent  to  99  percent  in  the  treatment  schools  and  from  47  percent  to  87  percent  in  the 
control  schools.  However,  the  12  significant  differences  between  the  treatment  and  control  schools  in  the  use 
of  SACD  activities  was  more  than  expected  by  chance  (which  would  be  3 out  of  62  comparisons),  and  in  all 
but  1 case  the  differences  favored  the  treatment  group.  This  may  reflect  the  fact  that  program  implementation 
and  program  training  for  staff  started  before  initial  data  collection. 

Impacts  on  Use  of  SACD  Activities 

The  introduction  of  the  formal  SS  program  would  be  expected  to  increase  the  use  of  SACD  activities  in  the 
treatment  schools  in  comparison  to  the  control  schools.  The  analysis  of  this  impact  was  based  on  the  Teacher 
Report  on  Classroom  and  School.  Every  spring,  third-,  fourth-,  and  fifth-grade  teachers  provided  information 
through  the  Teacher  Report  on  Classroom  and  School  about  the  social  and  character  development  activities 
they  used  in  their  classrooms.  Specifically,  information  from  the  Teacher  Report  on  Classroom  and  School 
was  used  to  determine  the  difference  between  treatment  and  control  teachers  in  these  areas: 

1 . the  use  of  SACD  activities  in  their  classrooms  overall  and  by  SACD  goal; 

2.  the  use  of  materials  and  strategies  to  implement  the  SACD  activities  within  classrooms  and  within 
the  entire  school; 

3.  the  use  of  staff  development  to  support  the  teachers;  and 

4.  teacher  support  for  SACD  efforts  in  the  school  and  the  use  of  practices  conducive  to  the  social  and 
character  development  of  students. 

Teacher  Report  on  Classroom  and  School  consent  and  completion  rates  (table  8.4)  led  to  87  percent  to  97 
percent  of  teachers  overall  having  data  for  the  3 years  (with  a greater  percentage  of  treatment  teachers 
providing  data  at  all  but  one  time  point).  To  estimate  intervention  impacts  for  each  of  the  outcomes,  testing 
of  the  statistical  significance  of  the  differences  in  means  was  used.  Preliminary  analysis  indicated  little  or  no 
gains  in  precision  from  using  covariates.  Before  testing  the  mean  differences,  the  data  were  weighted  such 
that  each  school  received  equal  weight.  Standard  errors  of  the  impact  estimates  account  for  the  clustering  of 
teachers  within  schools.  In  addition,  a set  of  heuristics  was  applied  to  determine  whether  each  outcome 
domain  was  statistically  significant  after  adjustments  were  made  for  the  multiple  tests  conducted. 

Use  of  Activities 

The  percentages  of  control  teachers  who  reported  using  any  SACD  activities  in  their  classrooms  ranged  from 
81  percent  to  92  percent  over  the  3 years  (table  8.13,  panel  1).  For  the  six  individual  SACD  goals,  the 
percentages  ranged  from  53  percent  to  86  percent  in  Year  1,  51  percent  to  89  percent  in  Year  2,  and  45 
percent  to  74  percent  in  Year  3.  Control  teachers’  use  of  behavior  management  activities  ranged  from  86 
percent  to  96  percent  over  this  period.  The  percentages  of  control  teachers  who  reported  using  any  SACD 
activities  in  their  classrooms  for  at  least  1 hour  per  week  (panel  2)  ranged  from  39  percent  to  74  percent  over 
the  3 years.  For  the  six  individual  SACD  goals,  the  percentages  ranged  from  0 percent  to  20  percent  in  Year 
1,  9 percent  to  23  percent  in  Year  2,  and  6 percent  to  23  percent  in  Year  3.  Control  teachers’  reported  use  of 
behavior  management  activities  ranged  from  67  percent  to  77  percent  over  this  period. 

For  teachers’  reported  use  of  any  SACD  activity  (panels  1 and  2),  48  comparisons  were  made,  with  2 expected 
to  be  significant  by  chance.  The  percentage  of  treatment  teachers  using  any  SACD  activity  was  not 
significantly  different  from  the  percentage  of  control  teachers  in  any  year.  There  were  significant  impacts  on 
the  use  of  specific  activities  in  all  3 years.  In  Year  1,  there  were  significant  impacts  on  violence  prevention  and 
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peace  promotion  (impact  = 31  percentage  points)  and  on  social  and  emotional  development  (impact  = 39 
percentage  points).  The  same  impacts  were  seen  in  Years  2 and  3 (impacts  ranging  from  25  to  40  percentage 
points).  In  Year  2,  there  was  a significant  impact  on  tolerance  and  diversity  (impact  = 23  percentage  points) 
and  another  in  Year  3 (impact  = 32  percentage  points).  In  Year  3,  there  was  an  impact  on  character  education 
(impact  = 22  percentage  points).  A similar  pattern  was  seen  in  the  reports  of  engagement  in  these  activities 
for  at  least  1 hour  per  week.  Significant  impacts  occurred  on  violence  prevention  and  peace  promotion  in 
Year  2 (impact  = 37  percentage  points),  social  and  emotional  development  activities  in  Year  1 (impact  = 37 
percentage  points)  and  Year  2 (impact  = 40  percentage  points),  character  education  in  Year  1 (impact  = 52 
percentage  points)  and  Year  2 (impact  = 37  percentage  points),  and  tolerance  and  diversity  in  Year  2 (impact 
= 26  percentage  points)  and  Year  3 (impact  =31  percentage  points).  The  use  of  any  activity  for  at  least  1 
hour  per  week  was  significantly  impacted  in  Year  1 (impact  = 30  percentage  points).  After  the  heuristics  were 
applied,  the  domain  for  engagement  in  SACD  activities  showed  that  SS  had  statistically  significant  impacts  for 
all  3 years. 

For  teachers’  reported  use  of  any  named  SACD  activity  (panels  3 and  4),  42  comparisons  were  made,  with  2 
expected  to  be  significant  by  chance.  Eight  of  the  14  impact  estimates  were  statistically  significant  in  Year  1, 
with  a significant  impact  on  the  domain  as  well.  Ten  of  the  14  tested  impacts  were  significant  in  Year  2,  and 
there  was  an  overall  impact  on  the  domain.  Seven  of  the  14  tested  impacts  were  significant  in  Year  3,  again 
with  a significant  impact  on  the  domain.  In  all  3 years,  impacts  were  seen  on  violence  prevention  (impact  = 
53,  69,  and  63  percentage  points),  social  and  emotional  development  (impact  = 73,  80,  and  81  percentage 
points),  character  education  (impact  = 45,  53,  and  61  percentage  points),  tolerance  and  diversity  (impact  = 49, 

45,  and  49  percentage  points),  and  any  named  activity  (impact  = 40,  49,  and  53  percentage  points).  For  any 
named  activity  at  least  1 hour  per  week,  SS  had  significant  impacts  on  violence  prevention  in  Year  2 (impact 
= 40  percentage  points),  social  and  emotional  development  in  Years  1 and  2 (impact  = 46  and  50  percentage 
points),  character  education  in  Years  1 and  2 (impact  = 46  and  45  percentage  points),  tolerance  and  diversity 
in  Years  2 and  3 (impact  = 27  and  28  percentage  points),  and  any  named  activity  in  all  3 years  (impact  = 49, 

46,  and  40  percentage  points).  After  the  heuristics  were  applied,  SS  had  statistically  significant  impacts  on  the 
domain  for  engagement  in  named  SACD  activities  in  all  3 years. 
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Table  8.13.  Impacts  on  teacher-reported  SACD  classroom  activities — SS 

Panel  1 : Engagement  in  any  activities  to  promote  SACD  goals1 

Year  1 Year  2 Year  3 

(Spring  3rd  grade) (Spring  4th  grade) (Spring  5th  grade) 

T reat-  T reat-  T reat- 


SACD  activity 

ment 

Control 

Impact 

p-value 

ment 

Control 

Impact 

p-value 

ment 

Control 

Impact 

p-value 

Teacher  sample  size 

61 

52 

60 

52 

59 

46 

Violence  prevention  and  peace  promotion 
(percent) 

87.3* 

56.2 

31.1 

0.017 

91.5* 

66.2 

25.3 

0.011 

88.5* 

50.5 

38.0 

0.005 

Social  and  emotional  development 
(percent) 

91.9* 

53.0 

38.9 

0.002 

96.7* 

64.9 

31.8 

0.002 

94.9* 

54.8 

40.1 

0.004 

Character  education  (percent) 

98.1 A 

86.4 

11.8 

0.083 

100.0 

88.8 

11.2 

t 

96.1* 

73.8 

22.3 

0.028 

Tolerance  and  diversity  (percent) 

80. 9A 

62.0 

19.0 

0.092 

85.1* 

62.3 

22.7 

0.002 

76.4* 

44.7 

31.8 

0.022 

Risk  prevention  and  health  promotion 
(percent) 

52.7 

63.2 

-10.4 

0.457 

57.9 

51.1 

6.8 

0.502 

59.3 

62.4 

-3.1 

0.819 

Civic  responsibility  and  community  service 
(percent) 

70.4 

73.7 

-3.3 

0.751 

67.3 

68.0 

-0.6 

0.959 

70.5 

66.7 

3.8 

0.732 

Any  SACD  goal  (percent) 

100.0 

87.8 

12.2 

t 

100.0 

91.5 

8.5 

t 

97. 9A 

80.7 

17.2 

0.076 

Behavior  management  (percent) 

89.4 

96.2 

-6.8 

0.187 

92.3 

85.7 

6.6 

0.213 

85.4 

89.8 

-4.4 

0.614 

See  notes  at  end  of  table. 
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Table  8.13.  Impacts  on  teacher-reported  SACD  classroom  activities — SS — Continued 

Panel  2:  Engagement  in  any  activities  to  promote  SACD  goals  for  at  least  1 hour  per  week 

Year  1 Year  2 Year  3 

(Spring  3rd  grade) (Spring  4th  grade) (Spring  5th  grade) 

T reat-  T reat-  T reat- 


SACD  activity 

ment 

Control 

Impact 

p-value 

ment 

Control 

Impact 

p-value 

ment 

Control 

Impact 

p-value 

Teacher  sample  size 

61 

52 

60 

52 

59 

46 

Violence  prevention  and  peace  promotion 
(percent) 

42.7 

0.0 

42.7 

t 

50.2* 

13.1 

37.1 

0.026 

38.2 

12.5 

25.7 

0.109 

Social  and  emotional  development 
(percent) 

49.8* 

12.4 

37.3 

0.007 

55.9* 

16.1 

39.8 

0.005 

48. 2A 

16.9 

31.3 

0.079 

Character  education  (percent) 

64.8* 

13.2 

51.5 

0.002 

60.2* 

23.0 

37.1 

0.040 

44.1 A 

23.3 

20.9 

0.094 

Tolerance  and  diversity  (percent) 

36.1 

19.6 

16.5 

0.199 

41.9* 

15.6 

26.2 

0.042 

38.5* 

7.4 

31.1 

0.010 

Risk  prevention  and  health  promotion 
(percent) 

13.9 

9.7 

4.2 

0.527 

13.3 

8.5 

4.8 

0.530 

16.0 

6.4 

9.6 

0.181 

Civic  responsibility  and  community  service 
(percent) 

11.2 

11.8 

-0.6 

0.934 

13.3 

17.0 

-3.7 

0.591 

1 3.9A 

10.6 

3.3 

0.699 

Any  SACD  goal  (percent) 

69.2* 

39.4 

29.7 

0.045 

83. 4A 

61.1 

22.2 

0.052 

85.4 

73.9 

11.5 

0.245 

Behavior  management  (percent) 

63.1 

76.6 

-13.5 

0.417 

79.3 

67.4 

11.9 

0.331 

73.0 

77.3 

-4.3 

0.688 

See  notes  at  end  of  table. 


Chapter  8.  Second  Step 


464 


Table  8.13.  Impacts  on  teacher-reported  SACD  classroom  activities — SS — Continued 

Panel  3:  Engagement  in  activities  to  promote  SACD  goals  linked  to  named  SACD  programs2 


Year  1 Year  2 Year  3 

(Spring  3rd  grade) (Spring  4th  grade) (Spring  5th  grade) 


SACD  activity 

T reat- 
ment 

Control 

Impact 

p-value 

T reat- 
ment 

Control 

Impact 

p-value 

T reat- 
ment 

Control 

Impact 

p-value 

Teacher  sample  size 

61 

52 

60 

52 

59 

46 

Violence  prevention  and  peace  promotion 
(percent) 

68.0* 

15.0 

53.0 

0.002 

76.3* 

7.3 

69.0 

0.001 

78.4* 

15.6 

62.8 

0.004 

Social  and  emotional  development 
(percent) 

86.8* 

13.9 

72.9 

0.001 

85.4* 

5.8 

79.6 

0.000 

93.4* 

12.8 

80.6 

0.001 

Character  education  (percent) 

85.9* 

41.4 

44.5 

0.008 

82.0* 

29.5 

52.6 

0.003 

85.1* 

24.4 

60.7 

0.003 

Tolerance  and  diversity  (percent) 

62.4* 

13.2 

49.2 

0.009 

55.2* 

10.2 

45.0 

0.000 

58.9* 

10.1 

48.8 

0.000 

Risk  prevention  and  health  promotion 
(percent) 

28.0 

21.6 

6.4 

0.520 

18.3 

12.8 

5.5 

0.487 

30.2 

15.6 

14.6 

0.267 

Civic  responsibility  and  community  service 
(percent) 

5.6 

0.0 

5.6 

T 

t 

t 

-3.8 

0.310 

13.8 

6.8 

7.0 

0.294 

Any  named  activity  (percent) 

97.0* 

57.3 

39.7 

0.001 

91.0* 

42.5 

48.5 

0.001 

97.9* 

45.0 

53.0 

0.004 

See  notes  at  end  of  table. 
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Table  8.13.  Impacts  on  teacher-reported  SACD  classroom  activities — SS — Continued 

Panel  4:  Engagement  in  activities  to  promote  SACD  goals  linked  to  named  SACD  programs  for  at  least  1 hour  per  week 

Year  1 Year  2 Year  3 

(Spring  3rd  grade) (Spring  4th  grade) (Spring  5th  grade) 

T reat-  T reat-  T reat- 


SACD  activity 

ment 

Control 

Impact 

p-value 

ment 

Control 

Impact 

p-value 

ment 

Control 

Impact 

p-value 

Teacher  sample  size 

61 

52 

60 

52 

59 

46 

Violence  prevention  and  peace  promotion 

(percent) 

42.7 

0.0 

42.7 

t 

t* 

t 

40.0 

0.002 

37.0 

8.7 

28.3 

0.138 

Social  and  emotional  development 

(percent) 

t* 

t 

46.1 

0.011 

t* 

t 

50.4 

0.002 

48. 2A 

9.1 

39.1 

0.083 

Character  education  (percent) 

57.3* 

11.1 

46.2 

0.002 

54.4* 

9.4 

45.0 

0.002 

42. 5A 

12.8 

29.7 

0.089 

Tolerance  and  diversity  (percent) 

36.1 

0.0 

36.1 

t 

t* 

t 

26.8 

0.005 

t* 

t 

28.1 

0.006 

Risk  prevention  and  health  promotion 

(percent) 

t 

t 

7.1 

0.187 

10.1 

7.3 

2.9 

0.688 

t 

t 

9.6 

0.119 

Civic  responsibility  and  community  service 

(percent) 

t 

4 

2.7 

t 

t 

t 

0.4 

0.904 

6.5 

0.0 

6.5 

t 

Any  named  activity  (percent) 

64.6* 

15.7 

49.0 

0.015 

59.5* 

13.6 

45.9 

0.001 

57.2* 

17.5 

39.7 

0.039 

t Not  applicable. 

% Reporting  standards  not  met.  Values  suppressed  to  protect  confidentiality. 

* Treatment  group  significantly  different  from  control  group  at  the  .05  level. 

A Treatment  group  significantly  different  from  control  group  at  the  .10  to  > .05  level. 

1 1n  all  3 years  at  least  one  outcome  remained  positive  and  statistically  significant  and  no  outcome  was  negative  and  statistically  significant  after  applying  the  Benjamini-Hochberg 
(1995)  procedure  to  adjust  significance  levels  downward  to  account  for  the  multiple  testing  of  impacts.  In  Year  2 at  least  half  of  the  impacts  were  positive  and  statistically  significant 
and  no  impact  was  negative  and  statistically  significant  based  on  univariate  statistical  tests. 

2 In  all  3 years  at  least  half  of  the  impacts  were  positive  and  statistically  significant  and  no  impact  was  negative  and  statistically  significant  based  on  univariate  statistical  tests  and  at 
least  one  outcome  remained  positive  and  statistically  significant  and  no  outcome  was  negative  and  statistically  significant  after  applying  the  Benjamini-Hochberg  (1995)  procedure  to 
adjust  significance  levels  downward  to  account  for  the  multiple  testing  of  impacts. 

NOTE:  Weights,  which  assign  equal  weight  to  each  school  within  the  program,  were  used  in  producing  the  treatment,  control,  and  overall  means. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Use  of  Materials  and  Strategies 

For  use  of  materials  and  strategies  to  support  SACD  goals,  87  comparisons  were  made,  with  4 expected  to  be 
significant  by  chance.  In  Year  1,  significant  impacts  were  found  for  teachers’  use  of  teacher  guides  (impact  = 
30  percentage  points),  instructional  aids  (impact  = 36  percentage  points),  role-playing  (impact  = 37 
percentage  points),  skill  training  (impact  = 59  percentage  points),  incorporating  SACD  into  the  academic 
curriculum  (impact  = 27  percentage  points),  presenting  role  models  (impact  =17  percentage  points),  using 
guided  visualization  (impact  = 43  percentage  points),  and  the  average  number  of  strategies  used  (by  2.7 
strategies  on  average).  In  Year  2,  SS  significantly  affected  the  use  of  teacher  guides  (impact  = 39  percentage 
points),  student  materials  (impact  = 30  percentage  points),  instructional  aids  (impact  = 24  percentage  points), 
direct  instruction  of  SACD  (impact  = 26  percentage  points),  skill  training  (impact  = 52  percentage  points), 
incorporating  SACD  into  academic  curriculum  (impact  = 21  percentage  points),  guided  visualization  (impact 
= 40  percentage  points),  and  average  number  of  strategies  used  (by  2.7  strategies  on  average).  Significant 
impacts  in  Year  3 affected  use  of  teacher  guides  (impact  = 45  percentage  points),  instructional  aids  (impact  = 
28  percentage  points),  skill  training  (impact  = 17  percentage  points),  parent  training  (impact  = 36  percentage 
points),  parent  and  community  involvement  (impact  = 25  percentage  points),  guided  visualization  (impact  = 
38  percentage  points),  and  average  number  of  strategies  (by  2.7  strategies  on  average).  The  SS  impact  on  the 
domain  of  materials  and  strategies  was  significant  in  all  3 years. 
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Table  8.14.  Impacts  on  use  of  SACD  classroom  materials  and  teaching  strategies — SS 


Year  1 

(Spring  3rd  grade) 

Year  2 

(Spring  4th  grade) 

Year  3 

(Spring  5th  grade) 

SACD  material  and  teaching  strategy1 

T reat- 
ment 

Control 

Impact 

p-value 

Treat- 

ment 

Control 

Impact 

p-value 

Treat- 

ment 

Control 

Impact 

p-value 

Teacher  sample  size 

61 

52 

60 

52 

59 
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Use  of  SACD  materials  (percent) 

Teacher  guides  (manuals,  curricula) 

93.0* 

62.7 

30.4 

0.000 

98.1* 

59.1 

39.0 

0.005 

94.7* 

50.2 

44.5 

0.003 

Student  materials  (workbooks  or  sheets) 

70.0 

52.1 

17.9 

0.126 

78.7* 

48.6 

30.1 

0.021 

67.1 

52.9 

14.3 

0.192 

Instructional  aids  (games,  software,  videos) 

64.6* 

28.7 

35.9 

0.005 

62.8* 

38.6 

24.2 

0.015 

61.5* 

33.6 

27.9 

0.005 

Giveaways  (bookmarks,  stickers) 

52.3 

66.6 

-14.3 

0.343 

59.1 

66.8 

-7.7 

0.504 

44.1 

62.3 

-18.2 

0.203 

Children’s  literature 

40.8 

35.4 

5.4 

0.765 

46.7 

38.7 

8.0 

0.587 

40.6 

33.9 

6.8 

0.381 

Other  types  of  materials 

8.5 

9.7 

-1.1 

0.843 

10.9 

12.2 

-1.2 

0.875 

t 

t 

4.7 

0.463 

Did  not  use  any  of  these  materials 

0.0 

13.2 

-13.2 

t 

0.0 

14.7 

-14.7 

t 

0.0 

14.1 

-14.1 

T 

Use  of  teaching  strategies  (percent) 

Role-playing 

97.5* 

60.2 

37.2 

0.008 

100.0 

62.9 

37.1 

t 

100.0 

73.1 

27.0 

t 

Cooperative  learning 

97.2 

92.9 

4.3 

0.365 

98.3 

95.5 

2.8 

0.404 

100.0 

91.7 

8.3 

t 

Peer  group  discussions 

95.4 

86.1 

9.3 

0.153 

91. 4A 

75.4 

16.0 

0.064 

100.0 

90.4 

9.6 

t 

Direct  instruction  of  SACD 

100.0 

72.8 

27.2 

t 

97.9* 

72.4 

25.5 

0.015 

100.0 

86.2 

13.8 

t 

Skill  training 

88.5* 

29.5 

59.1 

0.000 

89.0* 

36.7 

52.3 

0.000 

94.4* 

77.8 

16.5 

0.047 

Incorporating  SACD  into  academic 
curriculum 

96.3* 

69.1 

27.2 

0.003 

87.4* 

66.9 

20.5 

0.045 

94.0 

95.1 

-1.1 

0.825 

Parent  training 

20.1 

9.7 

10.5 

0.302 

t 

t 

2.4 

0.601 

t* 

t 

35.9 

0.002 

Parent/community  involvement 

41.7 

38.3 

3.5 

0.763 

36.1 A 

18.5 

17.6 

0.093 

62.2* 

37.5 

24.6 

0.020 

Mentoring 

45.7 

49.4 

-3.7 

0.820 

42.7 

42.5 

0.2 

0.988 

74.7 

67.6 

7.1 

0.569 

Good  behavior  notes  sent  home  daily 
or  weekly 

72.9 

76.8 

-3.9 

0.766 

73.6 

79.2 

-5.6 

0.626 

96.5 

91.5 

5.0 

0.219 

Presenting  role  models 

85.7* 

69.1 

16.5 

0.009 

87.6 

73.5 

14.1 

0.117 

92.5 

77.9 

14.6 

0.131 

See  notes  at  end  of  table. 
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Table  8.14.  Impacts  on  use  of  SACD  classroom  materials  and  teaching  strategies — SS — Continued 


Year  1 

Year  2 

Year  3 

(Spring  3rd  grade) 

(Spring  4th  grade) 

(Spring  5th  grade) 

SACD  material  and  teaching  strategy1 

Treat- 

ment 

Control 

Impact 

p-value 

T reat- 
ment 

Control 

Impact 

p-value 

T reat- 
ment 

Control 

Impact 

p-value 

Use  of  teaching  strategies  (percent) — 
Continued 

Targeted  story  reading  or  writing  on  social 
and  character  development  themes 

75.3 

60.1 

15.2 

0.266 

67.6 

55.9 

11.7 

0.303 

89.9 

85.9 

4.0 

0.614 

Peer  mediation 

38.8 

37.0 

1.9 

0.887 

38.0 

41.4 

-3.4 

0.822 

52.4 

49.4 

3.0 

0.833 

Honor  roll  for  positive  behavior 

38.6 

54.5 

-15.8 

0.371 

58.2 

53.0 

5.2 

0.749 

68.1 

57.0 

11.1 

0.554 

Pledges  or  recitations  on  social  and 
character  development  themes 

43.2 

29.2 

14.0 

0.451 

35.3 

32.1 

3.2 

0.838 

56.8 

40.9 

16.0 

0.317 

Guided  visualization 

73.9* 

30.8 

43.1 

0.000 

71.3* 

31.5 

39.9 

0.006 

92.0* 

54.6 

37.5 

0.001 

Student-led/student-assisted  instruction 

46.4 

38.3 

8.1 

0.553 

55.1 A 

33.3 

21.8 

0.051 

63.4 

71.9 

-8.5 

0.362 

Journaling 

68.7 

50.0 

18.7 

0.099 

67.4 

50.1 

17.2 

0.106 

84.3 

68.9 

15.4 

0.264 

Time  out  for  negative  behavior 

74.9 

86.1 

-11.2 

0.331 

82.9 

90.1 

-7.2 

0.185 

96.3 

97.1 

-0.8 

0.857 

Daily  or  weekly  rewards  for  positive 
behavior 

89.6 

80.0 

9.5 

0.192 

89.1 

88.6 

0.4 

0.949 

100.0 

97.2 

2.8 

t 

Any  strategy 

100.0 

98.6 

1.4 

t 

100.0 

100.0 

0.0 

t 

100.0 

97.2 

2.8 

t 

Number  of  strategies  (mean) 

13.8* 

11.1 

2.7 

0.001 

13.7* 

10.9 

2.7 

0.002 

16.3* 

13.6 

2.7 

0.000 

t Not  applicable. 

$ Reporting  standards  not  met.  Values  suppressed  to  protect  confidentiality. 

* Treatment  group  significantly  different  from  control  group  at  the  .05  level. 

A Treatment  group  significantly  different  from  control  group  at  the  .10  to  > .05  level. 

1 1n  all  3 years  at  least  one  outcome  remained  positive  and  statistically  significant  and  no  outcome  was  negative  and  statistically  significant  after  applying  the  Benjamini-Hochberg 
(1995)  procedure  to  adjust  significance  levels  downward  to  account  for  the  multiple  testing  of  impacts. 

NOTE:  Weights,  which  assign  equal  weight  to  each  school  within  the  program,  were  used  in  producing  the  treatment,  control,  and  overall  means. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Regarding  the  use  of  schoolwide  strategies,  18  comparisons  were  made  between  treatment  and  control 
teacher  reports,  with  1 expected  to  be  significant  by  chance.  There  was  no  SS  impact  on  teachers’  reports  of 
their  schools’  use  of  schoolwide  strategies  in  any  of  the  years  (these  data  are  not  shown  in  a table),  nor  was 
there  a significant  impact  on  the  domain  in  any  year. 

Participation  in  Professional  Development 

Regarding  reported  participation  in  professional  development,  27  comparisons  were  made  over  3 years,  with 
1 expected  to  be  significant  by  chance.  In  Year  1,  SS  had  an  impact  on  SACD  training  (impact  = 32 
percentage  points)  and  training  in  the  specific  goal  of  character  education  (impact  = 29  percentage  points). 
No  significant  impacts  were  found  on  the  domain  in  any  of  the  3 years. 
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Table  8.15.  Impacts  on  teacher-reported  SACD  professional  development — SS 


SACD  professional  development 

Year  1 

(Spring  3rd  grade) 

Year  2 

(Spring  4th  grade) 

Year  3 

(Spring  5th  grade) 

T reat- 
ment 

Control 

Impact 

p-value 

T reat- 
ment 

Control 

Impact 

p-value 

T reat- 
ment 

Control 

Impact 

p-value 

Teacher  sample  size 

61 

52 

60 

52 

59 
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SACD  training  in  past  12  months  (percent) 

87.0* 

54.7 

32.3 

0.009 

60.2 

48.1 

12.1 

0.295 

44.9 

42.8 

2.1 

0.864 

Hours  of  SACD  training  (mean) 

6.4 

4.5 

1.9 

0.302 

4.3 

4.8 

-0.4 

0.859 

3.1 

2.9 

0.1 

0.934 

Training  by  goal  (percent) 

Violence  prevention  and  peace  promotion 

21.0 

13.5 

7.5 

0.267 

10.6 

10.4 

0.2 

0.973 

12.9 

9.0 

3.9 

0.581 

Social  and  emotional  development 

32.1 

16.8 

15.3 

0.208 

14.4 

18.5 

-4.1 

0.690 

t 

t 

10.6 

0.140 

Character  education 

75.5* 

46.2 

29.3 

0.029 

43.7 

31.5 

12.2 

0.267 

24.4 

27.9 

-3.4 

0.792 

Tolerance  and  diversity 

9.7 

9.0 

0.7 

0.919 

13.5 

18.4 

-4.9 

0.647 

4.5 

8.3 

-3.8 

0.394 

Risk  prevention  and  health  promotion 

20.2 

25.2 

-5.0 

0.438 

8.6 

19.9 

-11.3 

0.178 

11.0 

14.6 

-3.6 

0.614 

Civic  responsibility  and  community  service 

t 

t 

5.0 

0.294 

t 

t 

5.0 

0.358 

t 

t 

-2.8 

0.502 

Behavior  management 

13.8 

27.9 

-14.1 

0.269 

16.2 

17.3 

-1.1 

0.891 

4.9A 

18.3 

-13.5 

0.073 

$ Reporting  standards  not  met.  Values  suppressed  to  protect  confidentiality. 

* Treatment  group  significantly  different  from  control  group  at  the  .05  level. 

A Treatment  group  significantly  different  from  control  group  at  the  .10  to  > .05  level. 

NOTE:  Weights,  which  assign  equal  weight  to  each  school  within  the  program,  were  used  in  producing  the  treatment,  control,  and  overall  means. 
SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Attitudes  and  Practices 

Teachers  reported  on  their  enthusiasm  for  SACD  efforts  in  their  schools  (these  data  are  not  shown  in  a table) 
by  indicating  enthusiasm,  cooperation,  or  open  dislike.  They  also  reported  on  the  SACD  practices  of  teachers 
and  staff  members  in  their  schools  (these  data  are  not  shown  in  a table).  The  practices  included  modeling 
positive  character  and  behavior  traits  with  students  and  fellow  teachers,  involving  students  in  making 
decisions,  giving  students  a voice  in  school  governance,  the  school  encouraging  parent  involvement  in 
children’s  social  and  character  development,  and  using  developmentally  appropriate  discipline  strategies 
rather  than  punishment  for  misbehavior.  Twenty-seven  comparisons  were  made  over  3 years,  with  1 expected 
to  be  significant  by  chance.  Out  of  9 comparisons  made  in  Year  2,  there  was  1 significant  impact  on  reported 
enthusiasm  (impact  = 24  percentage  points).  There  was  no  impact  on  the  domain  in  any  year. 

Year-by-Year  Impacts  on  Students  and  Perceptions  of  School  Climate 

The  primary  research  question  for  the  SS  evaluation  was  this: 

What  is  the  average  effect  of  the  SS  program  on  children’s  social  and  emotional  competence,  behavior, 
academics,  and  perceptions  of  school  climate? 

The  first  approach  to  answering  this  question  was  to  examine  the  year-by-year  impacts  of  the  SS  program  on 
these  student  and  school  climate  outcomes  over  the  3 years  as  the  students  progressed  from  third  through 
fifth  grades. 

Equation  (2)  (described  in  chapter  1)  was  estimated  to  provide  the  SS  impacts  on  the  20  outcomes  using  data 
from  the  12  treatment  and  control  schools.  For  the  SS  evaluation,  equation  (2)  excluded  the  program  fixed 
effects  (6p)  and  included  program-specific  covariates  and  random  school  effects  covariates.  Table  8.16  lists 
the  covariates  used  with  outcomes  from  each  report  in  the  SS  analysis. 
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Table  8.16.  Covariates  used  with  outcomes  from  each  report  for  analysis — SS 


Potential  covariate 

CR 

outcome 

PCR 

outcome 

TRS 

outcome 

TRCS 

outcome 

Total  number 

11 

28 

21 

8 

Child-reported 

Female 

V 

V 

✓ 

Hispanic 

V 

V 

✓ 

Black  (non-Hispanic) 

V 

V 

✓ 

Other  ethnicity 

V 

V 

✓ 

Age  in  years 

V 

V 

✓ 

Scales 

Afraid  at  School 

s 

✓ 

Altruistic  Behavior 

Empathy 

s 

✓ 

Engagement  with  Learning 

Negative  School  Orientation 

✓ 

✓ 

Normative  Beliefs  About  Aggression 

✓ 

✓ 

Sense  of  School  as  a Community 

V 

s 

✓ 

Problem  Behavior 

Self-Efficacy  for  Peer  Interactions 

Victimization  at  School 

Primary  caregiver-reported 

Age  in  years 

Completed  high  school  or  equivalent 

V 

s 

✓ 

Some  College 

V 

✓ 

Bachelor’s  or  higher  degree 

V 

✓ 

✓ 

Highest  level  of  education  in  household 

Completed  high  school  or  equivalent 

s 

Some  college 

Bachelor’s  or  higher  degree 

✓ 

Mother  present  in  home  life 

V 

Mother  and  father  present 

Respondent  someone  other  than  mother  or  father 

Number  of  people  in  household 

V 

Household  income:  $20,000  to  $40,000 

V 

Household  income:  $40,000  to  $60,000 

V 

Household  income:  More  than  $60,000 

V 

Income-to-poverty-threshold  ratio:  Below  135  percent 

Income-to-poverty-threshold  ratio:  135-185  percent 

Full-time  employment 

✓ 

Part-time  employment 

✓ 

See  notes  at  end  of  table. 
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Table  8.16.  Covariates  used  with  outcomes  from  each  report  for  analysis — SS — Continued 


Potential  covariate 

CR 

outcome 

PCR 

outcome 

TRS 

outcome 

TRCS 

outcome 

Parental  scales 

APQ-Poor  Monitoring  and  Supervision  Subscale 

APQ-Positive  Parenting  Subscale 

✓ 

✓ 

Child-Centered  Social  Control 

Confusion,  Hubbub,  and  Order 

Community  Resources 

Community  Risk 

✓ 

Parent  and  Teacher  Involvement 

Child  scales 

Altruistic  Behavior 

V 

✓ 

Positive  Social  Behavior 

✓ 

Problem  Behavior 

✓ 

✓ 

Teacher-reported 

Female 

✓ 

Hispanic 

✓ 

Black  (non-Hispanic) 

✓ 

Other  ethnicity 

✓ 

Total  teaching  experience 

✓ 

Total  experience  in  current  school 

Regular  certificate 

✓ 

Other  certificate 

✓ 

Highest  degree-bachelor’s 

✓ 

Child  scales 

Academic  Competence  and  Motivation 

✓ 

✓ 

✓ 

ADHD-Related  Behavior 

✓ 

Altruistic  Behavior 

✓ 

✓ 

Positive  Social  Behavior 

Problem  Behavior 

Parent  and  Teacher  Involvement 

NOTE:  Abbreviations  are 
CR:  Child  Report 
PCR:  Primary  Caregiver  Report 
TRS:  Teacher  Report  on  Student 
TRCS:  Teacher  Report  on  Classroom  and  School 
ADHD:  Attention  deficit  hyperactivity  disorder 
APC:  Alabama  Parenting  Cuestionnaire 
s : Covariate  used 
Blank  cell:  Covariate  not  used 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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To  assess  the  statistical  power  of  the  program -level  impact  estimates,  minimum  detectable  impacts  in  effect 
size  units  (MDES)  for  each  outcome  measure  were  calculated  for  the  SS  evaluation  (table  8.17).  MDES 
represent  the  smallest  impacts  in  effect  size  (standard  deviation)  units  that  can  be  detected  with  a high 
probability  (80%).  MDES  are  primarily  a function  of  study  sample  sizes,  the  degrees  of  freedom  available  for 
statistical  tests,  and  design  effects  due  to  clustering  (Schochet  2005).  For  the  SS  evaluation,  the  MDES  ranged 
from  0.099  to  0.526  for  the  child-level  outcomes  based  on  the  Child,  Caregiver,  and  Teacher  Report  on 
Student  and  from  0.314  to  0.694  for  the  school  climate  outcomes  based  on  the  Teacher  Report  on  Classroom 
and  School.  In  general,  the  MDES  for  the  school  climate  outcomes  were  larger  than  those  for  the  child-level 
outcomes. 
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Table  8.17.  Adjusted  minimum  detectable  effect  sizes  for  impact  evaluation — SS 


Outcome  measure-Report 

Year  1 

Year  2 

Year  3 

Social  and  Emotional  Competence  Domain 

Self-Efficacy  for  Peer  Interaction-CR 

0.156 

0.252 

0.105 

Normative  Beliefs  About  Aggression-CR 

0.103 

0.099 

0.099 

Empathy-CR 

0.156 

0.170 

0.200 

Behavior  Domain 

Altruistic  Behavior-CR 

0.103 

0.161 

0.133 

Altruistic  Behavior-PCR 

0.110 

0.132 

0.139 

Altruistic  Behavior-TRS 

0.526 

0.512 

0.481 

Positive  Social  Behavior-PCR 

0.142 

0.106 

0.108 

Positive  Social  Behavior-TRS 

0.279 

0.455 

0.356 

Problem  Behavior-CR 

0.106 

0.166 

0.099 

Problem  Behavior-PCR 

0.183 

0.142 

0.149 

Problem  Behavior-TRS 

0.275 

0.404 

0.348 

ADHD-Related  Behavior-TRS 

0.264 

0.296 

0.342 

Academics  Domain 

Engagement  with  Learning-CR 

0.103 

0.106 

0.099 

Academic  Competence  and  Motivation-TRS 

0.103 

0.167 

0.277 

Perceptions  of  School  Climate  Domain 

Positive  School  Orientation-CR 

0.192 

0.259 

0.270 

Negative  School  Orientation-CR 

0.201 

0.198 

0.123 

Student  Afraid  at  School-CR 

0.103 

0.197 

0.204 

Victimization  at  School-CR 

0.103 

0.185 

0.209 

Feelings  of  Safety-TRCS 

0.447 

0.550 

0.481 

Student  Support  for  Teachers-TRCS 

0.694 

0.314 

0.609 

NOTE:  Abbreviations  are 
CR:  Child  Report 
PCR:  Primary  Caregiver  Report 
TRS:  Teacher  Report  on  Student 
TRCS:  Teacher  Report  on  Classroom  and  School 
ADHD:  Attention  deficit  hyperactivity  disorder 
The  minimum  detectable  effect  (MDE)  formula  used  in  the  calculations  is  as  follows: 


MDE  = f actor (df)  * 


N 


Pi  (-  + -)  + (1  - Pi)  (—  + — ) 
\sr  sr/  \sTnr  srnr/ 


where  sT  and  sc  are  the  number  of  treatment  and  comparison  schools;  nT  and  nc  are  the  average  number  of  students  per 
classroom;  p,  is  the  intraclass  correlation  (ICC)  at  the  school  level;  and  factor(df)  is  a constant  that  depends  on  the  number  of 
degrees  of  freedom  (df)  available  for  analysis  (and  is  2.802  for  the  pooled  analysis). 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Table  8.18  provides  the  estimates  of  the  SS  impacts  on  each  of  the  20  outcomes  over  each  of  the  3 years  (60 
impacts  in  total,  with  3 expected  to  be  statistically  significant  by  chance).  Of  the  60  results,  1 was  statistically 
significant:  a detrimental  impact  was  found  on  fourth-graders’  Positive  Social  Behavior  (Primary  Caregiver 
Report,  effect  size  [ES]  = -0.14).  A substantively  important  but  nonsignificant  beneficial  impact  was  found  in 
all  3 years  for  Feelings  of  Safety  (Teacher  Report  on  Classroom  and  School,  ES  = 0.37,  0.39,  and  0.52). 
Application  of  the  heuristics  to  adjust  for  multiple  comparisons  within  each  outcome  domain  indicates  that 
SS  had  a statistically  significant  detrimental  impact  on  the  Social  and  Emotional  Competence  domain  in  Year  3. 
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Table  8.18.  Impacts  on  child  and  school  outcomes — SS 


Year  1 

(Spring  3rd  grade) 

Year  2 

(Spring  4th  grade) 

Year  3 

(Spring  5th  grade) 

Scale-Report 

T reat- 
ment 

Control 

Effect 
size  i 

o-value 

T reat- 
ment 

Control 

Effect 

size 

p-value 

T reat- 
ment 

Control 

Effect 

size 

p-value 

Social  and  Emotional  Competence  Domain1 

Self-Efficacy  for  Peer  Interactions-CR  (+) 

3.09 

3.03 

0.09 

0.337 

3.20 

3.24 

-0.08 

0.562 

3.24 

3.31 

-0.12 

0.146 

Normative  Beliefs  About  Aggression-CR  (-) 

1.22 

1.20 

0.06 

0.539 

1.23 

1.28 

-0.10 

0.228 

1.32 

1.26 

0.15 

0.130 

Empathy-CR  (+) 

2.38 

2.37 

0.04 

0.649 

2.25 

2.28 

-0.07 

0.463 

2.20 

2.26 

-0.14 

0.258 

Behavior  Domain 

Altruistic  Behavior-CR  (+) 

1.03 

1.10 

-0.09 

0.249 

0.89A 

1.00 

-0.18 

0.094 

0.88 

0.95 

-0.12 

0.184 

Altruistic  Behavior-PCR  (+) 

2.08 

2.10 

-0.03 

0.671 

2.06 

2.15 

-0.13 

0.144 

2.02' 

' 2.13 

-0.16 

0.098 

Altruistic  Behavior-TRS  (+) 

1.37 

1.40 

-0.08 

0.757 

1.30 

1.29 

0.03 

0.919 

1.27 

1.32 

-0.10 

0.675 

Positive  Social  Behavior-PCR  (+) 

3.18 

3.17 

0.03 

0.739 

3.20* 

3.28 

-0.14 

0.050 

3.30 

3.30 

0.01 

0.880 

Positive  Social  Behavior-TRS  (+) 

3.40 

3.32 

0.12 

0.252 

3.25 

3.36 

-0.17 

0.435 

3.28 

3.31 

-0.04 

0.800 

Problem  Behavior-CR  (-) 

0.12 

0.13 

-0.04 

0.659 

0.20 

0.17 

0.09 

0.406 

0.26 

0.22 

0.12 

0.201 

Problem  Behavior-PCR  (-) 

1.53 

1.52 

0.03 

0.727 

1.48 

1.50 

-0.05 

0.566 

1.47 

1.50 

-0.10 

0.280 

Problem  Behavior-TRS  (-) 

1.26 

1.27 

-0.02 

0.896 

1.29 

1.26 

0.10 

0.599 

1.33 

1.32 

0.02 

0.905 

ADHD-Related  Behavior-TRS  (-) 

1.59 

1.55 

0.07 

0.517 

1.60 

1.53 

0.13 

0.412 

1.57 

1.57 

-0.01 

0.969 

Academics  Domain 

Engagement  with  Learning-CR  (+) 

3.73 

3.74 

-0.03 

0.757 

3.70 

3.74 

-0.10 

0.210 

3.59 

3.64 

-0.12 

0.169 

Academic  Competence  and  Motivation- 

TRS  (+) 

3.34 

3.40 

-0.06 

0.161 

3.34A 

3.49 

-0.16 

0.055 

3.50 

3.32 

0.19 

0.137 

See  notes  at  end  of  table. 
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Table  8.18.  Impacts  on  child  and  school  outcomes — SS — Continued 


Year  1 

(Spring  3rd  grade) 

Year  2 

(Spring  4th  grade) 

Year  3 

(Spring  5th  grade) 

Scale-Report 

Treat- 

ment 

Control 

Effect 

size 

p-value 

T reat- 
ment 

Control 

Effect 

size 

p-value 

T reat- 
ment 

Control 

Effect 

size 

p-value 

Perceptions  of  School  Climate  Domain 
Positive  School  Orientation-CR  (+) 

3.13 

3.08 

0.09 

0.425 

2.91 

2.89 

0.05 

0.763 

2.83 

2.80 

0.06 

0.720 

Negative  School  Orientation-CR  (-) 

1.62 

1.65 

-0.06 

0.571 

1.73 

1.72 

0.03 

0.780 

1.87 

1.83 

0.08 

0.375 

Student  Afraid  at  School-CR  (-) 

1.93 

2.00 

-0.08 

0.332 

1.87 

1.95 

-0.10 

0.402 

1.86 

1.82 

0.05 

0.691 

Victimization  at  School-CR  (-) 

0.56 

0.62 

-0.09 

0.283 

0.55 

0.62 

-0.11 

0.306 

0.62 

0.60 

0.03 

0.833 

Feelings  of  Safety-TRCS  (+) 

4.19 

4.04 

0.37° 

0.216 

4.10 

3.86 

0.39° 

0.197 

4.02' 

' 3.63 

0.52° 

0.062 

Student  Support  for  Teachers-TRCS  (+) 

4.89 

4.11 

0.13 

0.663 

4.27 

4.22 

0.08 

0.678 

4.17 

4.12 

0.08 

0.768 

* Treatment  group  significantly  different  from  control  group  at  the  .05  level. 

A Treatment  group  significantly  different  from  control  group  at  the  .10  to  > .05  level. 
a Substantive  (but  nonsignificant  at  .05  level)  effect  size  of  2 .25  or  < -.25. 

1 Impact  on  domain  found  statistically  significant  and  detrimental  in  Year  3 based  on  the  fourth  heuristic  in  which  the  statistical  model  used  to  estimate  impacts  on  the  individual 
outcomes  was  re-estimated  using  a composite  of  all  the  outcome  variables  under  a domain.  The  domain  was  found  significant  if  the  impact  on  the  composite  was  significant.  The 
composite  was  formed  by  standardizing  each  outcome  variable  using  its  standard  deviation,  combining  the  values  of  the  outcome  variables,  and  taking  the  average  of  the  final  value. 
NOTE:  Abbreviations  are 
CR:  Child  Report 
PCR:  Primary  Caregiver  Report 
TRS:  Teacher  Report  on  Student 
TRCS:  Teacher  Report  on  Classroom  and  School 
ADHD:  Attention  deficit  hyperactivity  disorder 

The  +/-  signs  in  parentheses  indicate  the  direction  of  a beneficial  outcome.  All  impact  estimates  were  calculated  using  regression  models  where  each  program  and  school  within  a 
program  was  weighted  equally.  The  standard  errors  of  all  estimates  account  for  design  effects  due  to  unequal  weighting  and  the  clustering  of  students  within  schools.  See  table  1 .5  for 
information  about  the  measures  used  to  create  the  outcome  variables.  The  effect  size  was  calculated  by  dividing  the  estimated  impact  by  the  standard  deviation  of  the  outcome  measure 
for  the  control  group.  The  number  of  results  found  significant  was  no  more  than  expected  by  chance. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 


Chapter  8.  Second  Step 


Chapter  8.  Second  Step 


Impacts  on  Child  Outcomes  Over  Time 

SS  impacts  on  the  child  outcomes  over  time  were  estimated  using  growth  curve  models  by  examining 
treatment  and  control  group  differences  in  the  trajectories  of  student  outcomes  during  the  follow-up  period 
while  accounting  for  clustering  at  the  school  level.  The  growth  curve  models  are  estimated  using  a three-level 
hierarchical  linear  model,  where  Level  1 corresponds  to  time,  Level  2 to  students,  and  Level  3 to  schools 
(described  in  chapter  1). 

Table  8.19  provides  the  estimates  of  the  program’s  impacts  on  the  growth  in  student  outcomes  over  the  3 
years.  The  estimated  impacts  ranged  in  effect  size  units  (absolute  value)  from  0.00  to  0.19.  Three  of  the  18 
(with  1 expected  to  be  significant  by  chance)  estimated  SS  impacts  on  the  trajectories  of  child  outcomes  were 
statistically  significant  and  showed  a detrimental  impact  of  the  program  on  growth:  Empathy  (Child  Report, 
ES  = -0.13),  Positive  Social  Behavior  (Teacher  Report  on  Student,  ES  = -0.19),  and  Engagement  with 
Learning  (Child  Report,  ES  = -0.09). 
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Table  8.19.  Impacts  on  growth  of  child  outcomes — SS 


Scale-Report 

Average  growth  in  the  score  per  year1 

Mean  score  at 
implementation2 

Treatment 

Control 

Impact  on 
growth3 

Effect 

4 

size 

Standard 
error  of 
impact 

p-value  of 
impact 

Social  and  Emotional  Competence  Domain 

Self-Efficacy  for  Peer  Interactions-CR  (+) 

2.95 

0.12 

0.15 

-0.02 

-0.03 

0.03 

0.495 

Normative  Beliefs  About  Aggression-CR  (-) 

1.20 

0.05A 

0.01 

0.03 

0.07 

0.02 

0.090 

Empathy-CR  (+) 

2.43 

-0.11* 

-0.05 

-0.06 

-0.13 

0.02 

0.028 

Behavior  Domain 

Altruistic  Behavior-CR  (+) 

1.21 

-0.13 

-0.12 

-0.01 

-0.01 

0.04 

0.755 

Altruistic  Behavior-PCR  (+) 

2.14 

-0.05 

-0.02 

-0.04 

-0.04 

0.02 

0.159 

Altruistic  Behavior-TRS  (+) 

1.42 

-0.04 

-0.07 

0.03 

0.05 

0.06 

0.666 

Positive  Social  Behavior-PCR  (+) 

3.12 

0.05 

0.08 

-0.03 

-0.04 

0.01 

0.101 

Positive  Social  Behavior-TRS  (+) 

3.39 

-0.10* 

0.06 

-0.16 

-0.19 

0.06 

0.019 

Problem  Behavior-CR  (-) 

0.09 

0.06 

0.05 

0.00 

0.01 

0.01 

0.711 

Problem  Behavior-PCR  (-) 

1.53 

-0.02 

-0.02 

0.00 

0.00 

0.01 

0.848 

Problem  Behavior-TRS  (-) 

1.20 

0.08A 

0.01 

0.07 

0.17 

0.04 

0.086 

ADHD-Related  Behavior-TRS  (-) 

1.56 

0.02 

-0.02 

0.05 

0.06 

0.04 

0.285 

Academics  Domain 

Engagement  with  Learning-CR  (+) 

3.80 

-0.09* 

-0.04 

-0.05* 

-0.09 

0.02 

0.021 

Academic  Competence  and  Motivation-TRS  (+) 

3.35 

0.04 

0.03 

0.01 

0.01 

0.04 

0.808 

See  notes  at  end  of  table. 

Chapter  8.  Second  Step 


Table  8.19.  Impacts  on  growth  of  child  outcomes — SS — Continued 


Average  growth  in  the  score  per  year1 


Scale-Report 

Mean  score  at 
implementation2 

Treatment 

Control 

Impact  on 
growth3 

Effect 

4 

size 

Standard 
error  of 
impact 

p-value  of 
impact 

Perceptions  of  School  Climate  Domain 
Positive  School  Orientation-CR  (+) 

3.32 

-0.22 

-0.19 

-0.04 

-0.05 

0.04 

0.339 

Negative  School  Orientation-CR  (-) 

1.52 

0.14 

0.12 

0.02 

0.02 

0.02 

0.464 

Student  Afraid  at  School-CR  (-) 

2.08 

-0.12 

-0.09 

-0.03 

-0.03 

0.04 

0.403 

Victimization  at  School-CR  (-) 

0.60 

0.00 

0.00 

0.00 

0.00 

0.03 

0.930 

* Treatment  group  significantly  different  from  control  group  at  the  .05  level. 

A Treatment  group  significantly  different  from  control  group  at  the  .10  to  > .05  level. 

1 Pertains  to  the  estimated  slope  of  the  outcome  for  the  treatment  or  control  groups. 

2 The  average  score  at  implementation  is  calculated  across  treatment  and  control  groups,  using  regression  models  for  adjustment  on  covariates. 

3 Estimated  difference  between  the  slope  of  the  treatment  and  control  groups. 

4 Effect  size:  the  slope  of  the  treatment  group  minus  the  slope  of  the  control  group  divided  by  the  standard  deviation  of  the  outcome  for  the  program's  control  group  (the  standard 
deviation  is  calculated  without  accounting  for  school-level  clustering  or  regression  adjustments). 

NOTE:  Abbreviations  are 
CR:  Child  Report 
PCR:  Primary  Caregiver  Report 
TRS:  Teacher  Report  on  Student 
ADHD:  Attention  deficit  hyperactivity  disorder 

The  +/-  signs  in  parentheses  indicate  the  direction  of  a beneficial  outcome.  All  impact  estimates  were  calculated  using  HLM  6.06.  Sample  weights  were  used  in  all  analyses  to  give  (1) 
each  school  equal  weight  in  each  program  (within  each  time  period)  and  (2)  each  time  period  equal  weight  within  the  analysis.  See  table  1 .5  for  information  about  the  measures  used 
to  create  the  outcome  variables. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Summary 

As  part  of  the  Social  and  Character  Development  (SACD)  Research  Program,  researchers  at  the  Maryland  site 
evaluated  the  SS  program.  This  program  is  designed  to  promote  social  competence  and  reduce  children’s 
social  and  emotional  problems.  Twelve  public  schools  in  a single  school  district  in  Maryland  were  recruited  by 
the  Maryland  research  team  and  randomly  assigned  to  treatment  and  control  conditions  to  determine  the 
impact  of  SS  on  social  and  character  development  activities  in  the  schools  and  on  the  child  outcome  domains 
of  Social  and  Emotional  Competence,  Behavior,  Academics,  and  Perceptions  of  School  Climate. 

Analyses  of  the  initial  characteristics  of  the  sample  (students,  caregivers,  communities,  teachers,  and  schools) 
indicated  that  randomization  to  treatment  and  control  status  produced  groups  that  were  relatively  similar  at 
the  start  of  the  study  (with  1 out  of  84  comparisons  statistically  significantly  different,  fewer  than  would  be 
expected  by  chance).  The  data  on  the  initial  level  of  SACD  activity  led  to  two  findings.  First,  treatment 
teachers  reported  greater  use  of  and  training  in  SACD  activities  than  control  teachers,  and  they  did  so  more 
often  than  would  be  expected  by  chance  (11  out  of  62  comparisons,  with  3 expected  to  be  significant  by 
chance).  There  are  two  potential  causes  for  this  finding,  and  the  analysis  cannot  be  used  to  determine  whether 
the  reason  for  such  a difference  was  that  the  two  groups  did  differ  on  their  initial  use  of  SACD  activities  (i.e., 
that  randomization  did  not  create  similar  treatment  and  control  groups)  or  whether  the  fact  that  the  training 
of  all  treatment  teachers  and  the  implementation  of  the  SS  program  began  before  the  initial  data  were 
collected  (by  5 weeks)  influenced  the  teacher  reports.  Because  it  is  likely  (though  unproven)  that  the  training 
and  implementation  affected  the  teacher  reports,  these  data  were  not  considered  appropriate  for  use  as  a 
baseline  measure  of  SACD  activities  and  training  in  the  treatment  schools. 

Second,  these  data  indicate  that  the  control  condition  for  the  SACD  project  was  not  a “no  treatment”  control 
but  a “standard  practice”  control.  Because  the  control  teachers  were  not  affected  by  the  implementation  of 
the  SACD  programs  before  data  collection,  their  reports  reflect  standard  practice  in  the  control  schools. 
Standard  practice  at  the  control  schools  included  reports  of  47  percent  to  87  percent  of  teachers  using  any 
SACD  activities,  92  percent  of  teachers  using  specific  materials  in  conjunction  with  these  activities,  100 
percent  using  at  least  one  of  the  specified  instructional  strategies,  and  67  percent  participating  in  SACD 
training  over  the  past  12  months. 

Analyses  of  the  SS  impacts  on  level  of  SACD  activities  in  the  schools  revealed  impacts  on  the  use  of  SACD 
activities  (39  out  of  90)  and  materials  and  strategies  (23  out  of  87)  across  the  3 years.  There  were  impacts  on 
two  of  the  professional  development  activities  for  treatment  teachers  in  Year  1 (two  out  of  nine).  These  same 
measures  in  the  control  schools  across  the  3 years  of  the  study  confirmed  that  use  of  these  activities  in  the 
control  schools  constituted  their  standard  practice. 

Of  the  20  childdevel  outcomes  representing  the  four  domains  of  Social  and  Emotional  Competence, 
Behavior,  Academics,  and  Perceptions  of  School  Climate  assessed  in  each  of  the  3 years  of  the  study  (a  total 
of  60  results),  1 showed  a statistically  significant  detrimental  impact  of  the  SS  intervention.  A growth  curve 
analysis  was  used  to  analyze  the  change  over  time  in  these  same  outcomes  between  initial  data  collection  and 
the  final  outcome  data  collection  at  the  end  of  the  study.  Three  of  the  18  childdevel  outcomes  assessed 
showed  a significant  detrimental  impact  of  the  SS  program. 

The  SACD  evaluation  did  not  find  evidence  to  support  the  hypothesis  that  SS  had  beneficial  impacts  on 
students’  social  and  character  development.  Such  results  could  be  caused  by  the  inability  of  the  program  to 
cause  such  change,  possibly  because  the  theory  of  action  for  the  program  is  incomplete  or  the  activities  to 
carry  out  that  theory  are  not  effective. 

However,  these  results  may  also  be  due  to  the  inability  of  the  evaluation  to  observe  such  a change  due  to  the 
control  condition,  the  level  of  nonparticipation,  or  the  sample  size.  The  control  schools  continued  using  their 
standard  SACD  activities,  and  these  turned  out  to  be  high  in  quantity  and  broad  in  scope.  While  SS  had  a 
significant  positive  impact  on  the  amount  and  types  of  SACD  activities,  the  resulting  difference  in  the  amount 
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of  SACD  activities  between  the  treatment  and  control  schools  may  not  have  been  large  enough  to  cause 
significant  differences  in  the  student  outcomes.  Second,  29  percent  to  38  percent  of  the  students  in  the 
sample  universe  did  not  take  part  (depending  on  year)  because  of  nonconsent  or  noncompletion  of  the 
surveys.  As  a determination  could  not  be  made  as  to  whether  the  students  not  taking  part  differed 
significantly  from  those  who  did  take  part,  the  evaluation’s  results  are  valid  only  for  the  students  who  took 
part.  If  the  students  not  taking  part  were  different,  and  if  they  would  have  responded  better  to  SS  than  to  the 
SACD  activities  occurring  in  the  control  schools,  then  the  evaluation  could  have  underestimated  the 
program’s  impact.  Third,  the  sample  size  of  12  schools  and  the  resulting  higher  MDES  compared  to  those  for 
the  multiprogram  evaluation  may  have  reduced  the  likelihood  of  detecting  statistically  significant  effects. 
However,  it  should  be  noted  that  63  percent  of  the  MDES  for  the  60  outcomes  used  in  the  year -by-year 
analysis  were  below  0.25  (58%  were  below  0.20).  In  addition,  only  3 of  the  60  outcomes  were  found  to  be 
substantively  important;  however,  they  were  all  beneficial  and  referred  to  the  same  outcome  (Feelings  of 
Safety) . 


483 


This  page  is  intentionally  blank. 


References 


Achenbach,  T.M.  (1991).  Manual  for  Teacher’s  Report  Form  and  1991  Profile.  Burlington:  University  of  Vermont, 
Department  of  Psychiatry. 

American  Psychiatric  Association.  (2000).  Diagnostic  and  Statistical  Manual  of  Mental  Disorders  (4th  ed.,  revised 
text).  Washington,  DC:  American  Psychiatric  Association. 

Battistich,  V.,  Solomon,  D.,  Watson,  M.,  and  Schaps,  E.  (1997).  Caring  School  Communities.  Educational 
Psychologist,  32(3):  137-151. 

Benjamini,  Y.,  and  Hochberg,  Y.  (1995).  Controlling  the  False  Discovery  Rate — A New  and  Powerful 
Approach  to  Multiple  Testing  .Journal  of  the  Royal  Statistical  Society:  Series  B,  59:  289-300. 

Bennett,  K.J.,  Brown,  K.S.,  Boyle,  M.,  Racine,  Y.,  and  Offord,  D.  (2003).  Does  Low  Reading  Achievement  at 
School  Entry  Cause  Conduct  Problems?  Social  Science  and  Medicine,  54(11):  2443-2448. 

Brooks-Gunn,  J.,  and  Duncan,  G.J.  (1997).  The  Effects  of  Poverty  on  Children.  Children  and  Poverty,  7(2):  55- 

71. 

Bryk,  A.,  and  Raudenbush,  S.  (1992).  Hierarchical  Einear  Models  for  Social  and  Behavioral  Research:  Applications  and 
Data  Analysis  Methods.  Newbury  Park,  CA:  Sage. 

Carlson,  E.A.,  Sroufe,  L.A.,  Collins,  W.A.,  Jimerson,  S.,  Weinfield,  N.,  Hennighausen,  K.,  Egeland,  B.,  Hyson, 
D.M.,  Anderson,  F.,  and  Meyer,  S.E.  (1999).  Early  Environmental  Support  and  Elementary  School 
Adjustment  as  Predictors  of  School  Adjustment  in  Middle  Adolescence.  Journal  of  Adolescent  Research, 
14(1):  72-94. 

Caspi,  A.,  Taylor,  A.,  Moffitt,  T.E.,  and  Plomin,  R.  (2000).  Neighborhood  Deprivation  Affects  Children’s 
Mental  Health:  Environmental  Risks  Identified  in  a Genetic  Design.  Psychological  Science,  1 1:  338-342. 

Catalano,  R.F.,  Berglund,  M.L.,  Ryan,  J.A.M.,  Lonczak,  H.S.,  and  Hawkins,  J.D.  (2004).  Positive  Youth 
Development  in  the  United  States:  Research  Findings  on  Evaluations  of  Positive  Youth 
Development  Programs.  The  ANNAES  of  the  American  Academy  of  Political  and  Social  Science,  591  ( 1):  98- 

124. 

Conduct  Problems  Prevention  Research  Group  (CPPRG).  (1991).  Parent  Teacher  Involvement  Measure-Parent. 
Retrieved  from  http:/ / www.fasttrackproject.org/ techrept/ p/ ptp. 

Conduct  Problems  Prevention  Research  Group  (CPPRG).  (1999).  Initial  Impact  of  the  Fast  Track  Prevention 
Trial  for  Conduct  Problems  II:  Classroom  Effects,  journal  of  Consulting  and  Clinical  Psychology,  67(5): 
648-657. 

Crandall,  D.,  Loucks-Horsley,  S.,  Baucher,  J.,  Schmidt,  W.,  Eisman,  J.,  Cox,  P.,  Miles,  M.,  Huberman,  A., 
Taylor,  B.,  Goldberg,  J.,  Shive,  G.,  Thompson,  C.,  and  Taylor,  J.  (1982).  People,  Policies  and  Practices: 
Examining  the  Chain  of  School  Improvement,  Vols.  1-10.  Andover,  MA:  Network. 

Crosse,  S.,  Burr,  M.,  Cantor,  D.,  Hagen,  C.A.,  and  Hantman,  I.  (2001).  Wide  Scope,  Questionable  Quality:  Drug 
and  1/iolence  Prevention  Efforts  in  American  Schools:  Report  on  the  Study  on  School  1/iolence  and  Prevention. 
Washington,  DC:  Planning  and  Evaluation  Service,  U.S.  Department  of  Education. 

Dahlberg,  L.L.  (1998).  Youth  Violence  in  the  United  States:  Major  Trends,  Risk  Factors,  and  Prevention 
Approaches.  American  journal  of  Preventive  Medicine,  14(4):  259-272. 


485 


References 


Dahlberg,  L.L.,  and  Simon,  T.R.  (2006).  Predicting  and  Preventing  Youth  Violence:  Developmental  Pathways 
and  Risk.  InJ.  Lutzker  (Ed.),  Preventing  Violence:  Research  and  Evidence-Based  Intervention  Strategies  (pp.  97- 
124).  Washington,  DC:  American  Psychological  Association. 

Dahlberg,  L.L.,  Toal,  S.B.,  and  Behrens,  C.B.  (1998).  Measuring  Violence-Related  Attitudes,  Beliefs,  and  Behaviors 
Among  Youths:  A Compendium  of  Assessment  Tools.  Adanta,  GA:  Division  of  Violence  Prevention, 
National  Center  for  Injury  Prevention,  Centers  for  Disease  Control  and  Prevention. 

Damon,  W.  (1999).  The  Moral  Development  of  Children.  Scientific  American,  281(2):  72-79. 

Dempster,  A.P.,  Laird,  N.M.,  and  Rubin,  D.B.  (1977).  Maximum  Likelihood  Estimation  From  Incomplete 
Data  via  the  EM  Algorithm.  Journal  of  the  Royal  Statistical  Society:  Series  B,  39:  1-38. 

Dinkes,  R.,  Cataldi,  E.F.,  and  Lin-Kelly,  W.  (2007).  Indicators  of  School  Crime  and  Safety:  2007.  (NCES  2008021, 
NCJ  219553).  U.S.  Departments  of  Education  and  Justice.  Washington,  DC:  U.S.  Government 
Printing  Office. 

Dodge,  K.A.,  and  Coie,  J.D.  (1987).  Social-Information  Processing  Factors  in  Reactive  and  Proactive 
Aggression  in  Children’s  Peer  Groups.  Journal  of  Personality  and  Social  Psychology,  53:  1146-1 158. 

Dodge,  K.A.,  and  Pettit,  G.S.  (2003).  A Biopsychosocial  Model  of  the  Development  of  Chronic  Conduct 
Problems  in  Adolescence.  Developmental  Psychology,  39:  349-37 1 . 

Dunford,  F.W.,  and  Elliott,  D.S.  (1984).  Identifying  Career  Offenders  Using  Self-Reported  Data.  Journal  of 
Research  in  Crime  and  Delinquency,  21:  57-86. 

Eisenberg,  N.  (2000).  Emotion,  Regulation,  and  Moral  Development.  Annual  Review  of  Psychology,  51:  665-697. 

Emler,  N.  (1996).  How  Can  We  Decide  Whether  Moral  Education  Works?  Journal  of  Moral  Education,  25:  117- 
126. 

Farmer,  T.W.  (2000).  Social  Dynamics  of  Aggressive  and  Disruptive  Behavior  in  School:  Implications  for 
Behavior  Consultation.  Journal  of  Educational  and  Psychological  Consultation,  1 1:  299-322. 

Farmer,  T.W.,  Goforth,  J.B.,  Hives,  J.,  Aaron,  A.,  Hunter,  F.,  and  Sgmatto,  A.  (2006).  Competence 
Enhancement  Behavior  Management.  Preventing  School  Eailure,  50:  39-44. 

Farrell,  A.D.,  and  Camou,  S.  (2006).  School-Based  Interventions  for  Youth  Violence  Prevention.  In  R.J. 
Lutzker  (Ed.),  Preventing  Violence:  Research  and  Evidence-Based  Intervention  Strategies  (pp.  125-146). 
Washington,  DC:  American  Psychological  Association. 

Farrell,  A.D.,  and  Meyer,  A.L.  (1997).  The  Effectiveness  of  a School-Based  Curriculum  for  Reducing 
Violence  Among  Urban  Sixth-Grade  Students.  American  Journal  of  Public  Health,  87:  979-984. 

Farrington,  D.P.  (1989).  Early  Predictors  of  Adolescent  Aggression  and  Adult  Violence.  Violence  and  Victims, 
4:  79-100. 

Farrington,  D.P.,  and  Hawkins,  J.D.  (1991).  Predicting  Participation,  Early  Onset  and  Later  Persistence  in 
Officially  Recorded  Offending.  Criminal  Behavior  and  Mental  Health,  1:  1-33. 

Flannery,  D.J.,  Vazsonyi,  A.T.,  Liau,  A.K.,  Guo,  S.,  Powell,  K.E.,  Atha,  H.,  Vesterdal,  W.,  and  Embry,  D. 
(2003).  Initial  Behavior  Outcomes  for  the  PeaceBuilders  Universal  School-Based  Violence 
Prevention  Program.  Developmental  Psychology,  39:  292-308. 

Flay,  B.R.  (2002).  Positive  Youth  Development  Requires  Comprehensive  Health  Promotion  Programs. 
American  Journal  of  Health  Behavior,  26(6):  407-424. 

Flay,  B.R.,  Allred,  C.G.,  and  Ordway,  N.  (2001).  Effects  of  the  Positive  Action  Program  on  Achievement  and 
Discipline:  Two  Matched-Control  Comparisons.  Prevention  Science,  2:  71-89. 


486 


References 


Flay,  B.R.,  Graumlich,  S.,  Segawa,  E.  Bums,  J.L.,  and  Holliday,  M.Y.  (2004).  Effects  of  2 Prevention 
Programs  on  High-Risk  Behaviors  Among  African  American  Youth:  A Randomized  Trial.  Archives  of 
Pediatric  & Adolescent  Medicine,  158:  377-384. 

Forehand,  R.,  Brody,  G.,  Armistead,  L.,  Dorsey,  S.,  Morse,  E.,  Morse,  P.,  and  Stock,  M.  (2000).  The  Role  of 
Community  Risks  and  Resources  in  the  Psychosocial  Adjustment  of  At-Risk  Children:  An 
Examination  Across  Two  Community  Contexts  and  Two  Informants.  Behavior  Therapy,  31:  395-414. 

Fors,  S.W.,  Crepaz,  N.,  and  Hayes,  D.M.  (1999).  Key  Factors  That  Protect  Against  Health  Risks  in  Youth: 
Further  Evidence.  American  Journal  of  Health  Behavior,  23 : 368-380. 

Fraser,  M.W.,  Nash,  J.K.,  Galinsky,  M.J.,  and  Darwin,  K.E.  (2000).  Making  Choices:  Social  Problem-Solving  Skills 
for  Children.  Washington,  DC:  NASW  Press. 

Funk,  J.,  Elliott,  R.,  Bechtoldt,  H.,  Pasold,  T.,  and  Tsavoussis,  A.  (2003).  The  Attitudes  Toward  Violence 
Seal  c.  Journal  of  Interpersonal  Violence,  18:  186-196. 

Furrer  C.,  and  Skinner,  E.  (2003).  Sense  of  Relatedness  as  a Factor  in  Children’s  Academic  Engagement  and 
Performance.  Journal  of  educational  Psychology , 95:  148-162. 

Gershoff,  E.T.,  and  Aber,  J.L.  (2006).  Neighborhoods  and  Schools:  Contexts  and  Consequences  for  the 
Mental  Health  and  Risk  Behaviors  of  Children  and  Youth.  In  L.  Balta  and  C.  Tamis-LeMonda  (Eds.), 
Child  Psychology:  A Handbook  of  Issues  (2nd  ed.)  (pp.  611-645).  New  York:  Psychology  Press/Taylor  & 
Francis. 

Gorman-Smith,  D.,  Tolan,  P.H.,  Henry,  D.B.,  and  Florsheim,  P.  (2000).  Patterns  of  Family  Functioning  and 
Adolescent  Outcomes  Among  Urban  African  American  and  Mexican  American  Families.  Journal  of 
Family  Psychology,  14:  436-457. 

Gorman-Smith,  D.,  Tolan,  P.H.,  Zelli,  A.,  and  Huesmann,  L.R.  (1996).  The  Relation  of  Family  Functioning  to 
Violence  Among  Inner-City  Minority  Youths.  Journal  of  Family  Psychology,  10:  115-129. 

Gottfredson,  G.D.,  and  Gottfredson,  D.C.  (2001).  What  Schools  Do  to  Prevent  Problem  Behavior  and 
Promote  Safe  Environments.  Journal  of  educational  and  Psychological  Consultation,  12:  313-344. 

Greenberg,  M.T.  (2004).  Current  and  Future  Challenges  in  School -Based  Prevention:  The  Researcher 
Perspective.  Prevention  Science,  5:  5-13. 

Greenberg,  M.T.,  Weissberg,  R.P.,  Utne  O’Brian,  M.,  Zins,  J.E.,  Fredericks,  L.,  Resnik,  H.,  and  Elias,  M.J. 
(2003).  Enhancing  School-Based  Prevention  and  Youth  Development  Through  Coordinated  Social, 
Emotional,  and  Academic  Learning.  American  Psychologist,  58:  466-474. 

Gresham,  F.M.,  and  Elliott,  S.N.  (1990).  Social  Skills  Rating  System.  Circle  Pines,  MN:  American  Guidance 
Service. 

Griffith,  J.  (1996).  Relation  of  Parental  Involvement,  Empowerment,  and  School  Traits  to  Student  Academic 
Performance.  Journal  of  educational  Research,  90(1):  33-41. 

Grolnick,  W.S.,  Ryan,  R.M.,  and  Deci,  E.L.  (1991).  Inner  Resources  for  Student  Motivation:  Motivational 
Mediators  of  Children’s  Perceptions  of  Their  Parents.  Journal  of  educational  Psychology,  83:  508-517. 

Grossman,  D.,  Neckerman,  H.J.,  Koepsell,  T.D.,  Liu,  P.,  Asher,  K.,  Beland,  K.,  Frey,  K.,  and  Rivara,  F.P. 
(1997).  Effectiveness  of  a Violence  Prevention  Curriculum  Among  Children  in  Elementary  School:  A 
Randomized  Controlled  Trial.  Journal  of the  American  Medical  Association,  277:  1605-1611. 

Guerra,  N.G.,  Huessman,  L.R.,  Tolan,  P.H.,  VanAcker,  R.  and  Eron,  L.D.  (1995).  Stressful  Events  and 
Individual  Beliefs  as  Correlates  of  Economic  Disadvantage  and  Aggression  Among  Urban  Children. 
Journal  of  Consulting  and  Clinical  Psychology,  63:  518-528. 

Hahn,  R.,  Fuqua-Whitley,  D.,  Wethington,  H.,  Lowy,  J.,  Crosby,  A.,  Fullilove,  M.,  Johnson,  R.,  Liberman,  A., 
Moscicki,  E.,  Price,  L.,  Snyder,  S.,  Tuma,  F.,  Cory,  S.,  Stone,  G.,  Mukhopadhaya,  K.,  Chattopadhyay, 


487 


References 


S.,  and  Dahlberg,  L.  (2007).  Effectiveness  of  Universal  School-Based  Programs  to  Prevent  Violent 
and  Aggressive  Behavior:  A Systematic  Review.  American  Journal  of  Preventive  Medicine,  33{ 2S):  SI  14- 
S129. 

Harris,  R.J.  (1975).  A Primer  of  Multivariate  Statistics.  New  York:  Academic  Press. 

Henry,  D.B.,  Tolan,  P.H.,  and  Gorman-Smith,  D.  (2001).  Longitudinal  Family  and  Peer  Group  Effects  on 
Violent  and  Non-Violent  Delinquency.  Journal  of  Child  Clinical  Psychology,  16:  203-220. 

Huesmann,  L.R.,  and  Guerra,  N.G.  (1997).  Children’s  Normative  Beliefs  About  Aggression  and  Aggressive 
Behavior.  Journal  of  Personality  and  Social  Psychology,  72 : 408-419. 

Joreskog,  K.G.,  Sorbom,  D.,  Du  Toit,  S.,  and  Du  Toit,  M.  (2003).  EISREE  8:  New  Statistical  Features  (3rd 
printing,  with  revisions).  Lincolnwood,  IL:  Scientific  Software  International. 

Kazdin,  A.E.  (2001).  Behavior  Modification  in  Applied  Settings  (6th  ed.).  Belmont,  CA:  Wadsworth/Thomson 
Learning. 

Kellam,  S.G,  Ling,  X.,  Merisca,  R.,  Brown,  C.H.,  and  lalong,  N.  (1998).  The  Effect  of  the  Level  of  Aggression 
in  the  First  Grade  Classroom  on  the  Course  and  Malleability  of  Aggressive  Behavior  into  Middle 
School.  Development  and  Psychopathology,  10:  165-185. 

Kusche,  C.A.,  and  Greenberg,  M.T.  (1994).  The  PATHS  Curriculum.  Seatde,  WA:  Developmental  Research 
and  Programs. 

Larson,  R.  W.  (2000).  Toward  a Psychology  of  Positive  Youth  Development.  American  Psychologist,  55:  170- 
183. 

Lewis,  L.,  Parsad,  B.,  Carey,  N.,  Bartfai,  N.,  Farris,  E.,  and  Smerdon,  B.  (1999).  Teacher  Quality:  A Report  on  the 
Preparation  and  Qualifications  of  Public  School  Teachers  (NCES  1999-080).  Washington,  DC:  U.S. 
Department  of  Education. 

Lickona,  T.  (1993).  The  Return  of  Character  Education.  Educational  leadership,  51:  6-11. 

Limber,  S.P.,  and  Small,  M.A.  (2003).  State  Laws  and  Policies  to  Address  Bullying  in  Schools.  School  Psychology 
Review,  32:  445-455. 

Loeber,  R.,  and  Stouthamer-Loeber,  M.  (1998).  Juvenile  Aggression  at  Home  and  at  School.  In  D.S.  Elliott, 
B.A.  Hamburg,  and  K.R.  Williams  (Eds.),  Violence  in  American  Schools  (pp.  94-126).  Cambridge: 
Cambridge  University  Press. 

Loney,  J.,  and  Milich,  R.  (1982).  Hyperactivity,  Inattention,  and  Aggression  in  Clinical  Practice.  In  M. 
Wolraich  and  D.  Routh  (Eds.),  Advances  in  Developmental  and  Behavioral  Pediatrics  (Vol.  2,  pp.  113-147). 
Greenwich,  CT:  JAI  Press. 

Malecki,  C.K.,  and  Elliot,  S.N.  (2002).  Children’s  Social  Behaviors  as  Predictors  of  Academic  Achievement:  A 
Longitudinal  Analysis.  School  Psychology  Quarterly,  17:  1-23. 

Mansfield,  W.,  Alexander,  D.,  and  Farris,  E.  (1991).  Fast  Response  Survey  System,  Teacher  Survey  on  Safe,  Disciplined, 
and  Drug-Free  Schools  (NCES  91-091).  National  Center  for  Education  Statistics,  Office  of  Educational 
Research  and  Improvement,  U.S.  Department  of  Education.  Washington,  DC:  U.S.  Government 
Printing  Office. 

Matheny,  A.P.,  Wachs,  T.D.,  Ludwig,  J.L.,  and  Phillips,  K.  (1995).  Bringing  Order  Out  of  Chaos: 
Psychometric  Characteristics  of  the  Confusion,  Hubbub,  and  Order  Scale.  Journal  of  Applied 
Developmental  Psychology,  16:  429-444. 

McCord,  J.,  Widom,  C.S.,  Barnba,  M.I.,  and  Crowell,  N.A.  (2000).  Education  and  Delinquency:  Summary  of  a 
Workshop.  Washington,  DC:  National  Academy  of  Sciences. 


488 


References 


McEvoy,  A.,  and  Welker,  R.  (2000).  Antisocial  Behavior,  Academic  Failure,  and  School  Climate:  A Critical 
Review . Journal  of  Emotional  and  behavioral  Disorders,  8:  130-141. 

McMahon,  R.J.,  and  Frick,  P.J.  (2005).  Evidence-Based  Assessment  of  Conduct  Problems  in  Children  and 
Adolescents.  Journal  of  Clinical  Child  and  Adolescent  Psychology,  34,  477-505. 

Merrell,  K.,  Gueldner,  B.A.,  Ross,  S.W.,  and  Isava,  D.M.  (2008).  How  Effective  Are  School  Bullying 
Intervention  Programs?  A Meta-Analysis  of  Intervention  Research.  School  Psychology  Quarterly,  23\  26- 
42. 

Moffit,  T.E.  (1993).  Adolescence-Limited  and  Life-Course-Persistent  Antisocial  Behavior:  A Developmental 
Taxonomy.  Psychological  Review,  100:  674-701. 

Murray,  D.  (1998).  Design  and  Analysis  of  Group-Randomised  Trials.  Oxford:  Oxford  University  Press. 

Mytton,  J.,  DiGuiseppi,  C.,  Gough,  D.,  Taylor,  R.,  and  Logan,  S.  (2006).  School-Based  Secondary  Prevention 
Programmes  for  Preventing  Violence.  Cochrane  Database  of  Systematic  Reviews,  Issue  3.  doi: 

1 0.1 002/ 14651 858.CD004606.pub2. 

Najaka,  S.S.,  Gottfredson,  D.C.,  and  Wilson,  D.B.  (2001).  A Meta-Analytic  Inquiry  Into  the  Relationship 
Between  Selected  Risk  Factors  and  Problem  Behavior.  Prevention  Science,  2\  257-271. 

O’Donnell,  J.,  Hawkins,  J.D.,  and  Abbott,  R.D.  (1995).  Predicting  Serious  Delinquency  and  Substance  Use 
Among  Aggressive  Boys.  Journal  of  Consulting  and  Clinical  Psychology,  63\  529-537. 

Orpinas,  P.,  and  Frankowski,  R.  (2001).  The  Aggression  Scale:  A Self-Report  Measure  of  Aggressive  Behavior 
for  Young  Adolescents.  Journal  of  Early  Adolescence,  21(f):  50-67. 

Orpinas,  P.,  and  Horne,  A.M.  (2006).  bullying  Prevention:  Creating  a Positive  School  Climate  and  Developing  Social 
Competence.  Washington,  DC:  American  Psychological  Association. 

Orpinas,  P.,  and  Kelder,  S.  (1995).  Students  for  Peace  Project:  Second  Student  Evaluation  (Unpublished  manuscript). 
University  of  Texas  Health  Science  Center  at  Houston,  School  of  Public  Health,  Houston. 

Park-Higgerson,  H.-K.,  Perumean-Chaney,  S.E.,  Bartolucci,  A.A.,  Grimley,  D.M.,  and  Singh,  K.P.  (2008).  The 
Evaluation  of  School-Based  Violence  Prevention  Programs:  A Meta-Analysis.  Journal  of  School  Health, 
78:  465-479. 

Pelham,  W.E.,  Fabiano,  G.A.,  and  Massetti,  G.M.  (2005).  Evidence-Based  Assessment  of  Attention  Deficit 
Hyperactivity  Disorder  in  Children  and  Adolescents.  Journal  of  Clinical  Child  and  Adolescent  Psychology, 
34:  449-476. 

Pelham,  W.E,  Gnagy,  E.M.,  Greenslade,  K.E.,  and  Milich,  R.  (1992).  Teacher  Ratings  of  DSM-III-R 
Symptoms  for  the  Disruptive  Behavior  Disorders.  Journal  of  the  American  Academy  of  Child  and 
Adolescent  Psychiatry,  31:  210-218. 

Rentoul,  A.J.,  and  Fraser,  B.J.  (1983).  Development  of  a School-Level  Environment  Questionnaire.  Journal  of 
Educational  Administration,  21(f):  21—39. 

Rey,  J.M.,  Sawyer,  M.G.,  and  Prior,  M.R.  (2005).  Similarities  and  Differences  Between  Aggressive  and 
Delinquent  Children  and  Adolescents  in  a National  Sample.  Australian  and  New  Zealand  Journal  of 
Psychiatry,  39:  366-372. 

Reynolds,  C.R.,  and  Kamphaus,  R.W.  (1998).  behavioral  Assessment  System  for  Children.  Circle  Pines,  MN: 
American  Guidance  Service,  Inc. 

Roberts,  W.,  Horn,  A.,  and  Battistich,  V.  (1995).  Assessing  Students’  and  Teachers’  Sense  of  School  as  a 
Caring  Community.  Paper  presented  at  the  Annual  Meeting  of  the  American  Educational  Research 
Association,  San  Francisco,  CA. 

Rubin,  D.  (1987).  Multiple  Imputation  for  Nonresponse  in  Surveys.  New  York:  John  Wiley  & Sons,  Inc. 


489 


References 


Samples,  F.,  and  Aber,  L.  (1998).  Evaluations  of  School-Based  Violence  Prevention  Programs.  In  D.S.  Elliott 
and  B.A.  Hamburg  (Eds.),  Violence  in  American  Schools:  A New  Perspective  (pp.  217-252).  Cambridge: 
Cambridge  University  Press. 

Sampson,  R.J.  (2000).  A Neighborhood-Level  Perspective  on  Social  Change  and  the  Social  Control  of 
Adolescent  Delinquency.  In  L.J.  Crocket  and  R.K.  Silbereisen  (Eds.),  Negotiating  Adolescence  in  Times  of 
Social  Change  (pp.  178-188).  New  York:  Cambridge  University  Press. 

Sampson,  R.J.  Morenoff,  J.,  and  Earls,  F.  (1999).  Beyond  Social  Capital:  Spatial  Dynamics  of  Collective 
Efficacy  for  Children.  American  Sociological  Review,  64:  633-660. 

Schochet,  P.  (2005).  Statistical  Power  for  Random  Assignment  Evaluations  of  Education  Programs.  Princeton,  NJ: 
Mathematica  Policy  Research. 

Schochet,  P.Z.  (2008a).  Technical  Methods  Report.  Guidelines  for  Multiple  Testing  in  Impact  Evaluations  (NCEE  2008- 
4018).  Washington,  DC:  National  Center  for  Education  Evaluation  and  Regional  Assistance,  Institute 
of  Education  Sciences,  U.S.  Department  of  Education. 

Schochet,  P.Z.  (2008b).  The  Eate  Pretest  Problem  in  Randomised  Control  Trials  of  Education  Interventions  (NCEE 
2009-4033).  Washington,  DC:  National  Center  for  Education  Evaluation  and  Regional  Assistance, 
Institute  of  Education  Sciences,  U.S.  Department  of  Education. 

Schultz,  D.,  Izard,  C.,  and  Bear,  G.  (2004).  Children’s  Emotion  Processing:  Relations  to  Emotionality  and 
Aggression.  Development  and  Psychopathology,  16:  371-187. 

Schwartz,  D.,  and  Proctor,  L.J.  (2000).  Community  Violence  Exposure  and  Children’s  Social  Adjustment  in 
the  School  Peer  Group:  The  Mediating  Roles  of  Emotion  Regulation  and  Social  Cognition.  Journal  of 
Consulting  and  Clinical  Psychology,  68:  670-683. 

Seligman,  M.E.,  and  Csikszentmihalyi,  M.  (2000).  Positive  Psychology:  An  Introduction.  American  Psychologist, 
56(1):  5-13. 

Shaver,  V.,  and  Walls,  R.T.  (1998).  Effect  of  Title  I Parent  Involvement  on  Student  Reading  and  Mathematics 
Achievement.  Journal  of  Research  and  Development  in  Education , 31:  90-97. 

Shelton,  K.K.,  Frick,  P.J.,  and  Wootton,  J.  (1996).  Assessment  of  Parenting  Practices  in  Families  of 
Elementary  School-Age  Children.  Journal  of  Clinical  Child  Psychology,  25:  317-329. 

Slaby,  R.G.,  and  Guerra,  N.G.  (1988).  Cognitive  Mediators  of  Aggression  in  Adolescent  Offenders:  1. 
Assessment.  Developmental  Psychology,  24:  580-588. 

Smith,  B.H.,  Molina,  B.S.G.,  Massetti,  G.M.,  Waschbusch,  D.A.,  and  Pelham,  W.E.  (2007).  School-Wide 
Interventions:  The  Foundation  of  a Public  Health  Approach  to  School-Based  Mental  Health.  In  S.W. 
Evans,  M.  Weist,  and  Z.  Serpell  (Eds.),  Advances  in  School-Based  Mental  Health  Interventions:  Best  Practices 
and  Program  Models  (Vol.  2,  pp.  7-2—7-19).  Kingston,  NJ:  Civic  Research  Institute 

Solomon,  D.,  Battistich,  V.,  Watson,  M.,  Schaps,  E.,  and  Lewis,  C.  (2000).  A Six-District  Study  of  Educational 
Change:  Direct  and  Mediated  Effects  of  the  Child  Development  Project.  Social  Psychology  of  Education, 
4(1):  3-51. 

Stage,  S.A.,  and  Quiroz,  D.R.  (1997).  A Meta-Analysis  of  Interventions  to  Decrease  Disruptive  Classroom 
Behavior  in  Public  Education  Settings.  School  Psychology  Review,  26(3):  333-368. 

Stringfield,  S.,  Millsap,  M.,  Herman,  R.,  Yoder,  N.  Brigham,  N.,  Nesselrodt,  P.,  Schaffer,  E.,  Karweit,  N. 
Levin,  M.,  and  Stevens,  R.  (1997).  Urban  and  Suburban  I Rural  Special  Strategies  for  Educating  Disadvantaged 
Children:  Pinal  Report.  Washington,  DC:  U.S.  Department  of  Education. 


490 


References 


Takeuchi,  D.,  Williams,  D.R.,  and  Adair,  R.  (1991).  Economic  Stress  and  Children’s  Emotional  and 
Behavioral  Problems.  Journal  of  Marriage  and  the  Family,  53:  1031-1041. 

Tolan,  P.H.,  Guerra,  N.G.,  and  Kendall,  P.C.  (1995).  A Developmental-Ecological  Perspective  on  Antisocial 
Behavior  in  Children  and  Adolescents:  Toward  a Unified  Risk  and  Intervention  Framework.  Journal  of 
Consulting  and  Clinical  Psychology,  63\  579-584. 

Thornberry,  T.P.,  Huizinga,  D.,  and  Loeber,  R.  (2004).  The  Causes  and  Correlates  Studies:  Findings  and 
Policy  Implications  .Juvenile  Justice,  9:  3-19. 

Trzesniewski,  K.H.,  Moffitt,  T.E.,  Caspi,  A.,  Taylor,  A.,  and  Maughan,  B.  (2006).  Revisiting  the  Association 
Between  Reading  Achievement  and  Antisocial  Behavior:  New  Evidence  of  Environmental 
Explanation  From  a Twin  Study.  Child  Development,  77:  72-88. 

U.S.  Department  of  Education,  National  Center  for  Education  Statistics,  Common  Core  of  Data.  (2003-04). 
Data  retrieved  from  http:/ / nces.ed.gov/ ccd. 

Walker,  H.M.,  and  Shinn,  M.R.  (2002).  Structuring  School-Based  Interventions  to  Achieve  Integrated 
Primary,  Secondary,  and  Tertiary  Prevention  Goals  for  Safe  and  Effective  Schools  (pp.  1-26).  In  M.R. 
Shin,  H.M.  Walker,  and  G.  Stoner  (Eds.),  Interventions  for  Academic  and  Behavioral  Problems  II:  Preventive 
and  Remedial  Approaches.  Bethesda,  MD:  National  Association  of  School  Psychologists. 

Waschbusch,  D.,  Pelham,  W.,  and  Massetti,  G.  (2005).  The  Behavior  Education  Support  and  Treatment 
(BEST)  School  Intervention  Program:  Pilot  Project  Data  Examining  Schoolwide,  Targeted-School, 
and  Targeted-Home  Approaches.  Journal  of  Attention  Disorders,  9:  313-322. 

Weissberg,  R.P.,  Kumpfer,  K.L.,  and  Seligman,  M.E.P.  (2003).  Special  Issue:  Prevention  That  Works  for 
Children  and  Youth:  An  Introduction.  American  Psychologist,  58:  425-432. 

Wentzel,  K.R.  (1991).  Relations  Between  Social  Competence  and  Academic  Achievement  in  Early 
Adolescence.  Child  Development,  62:  1066-1078. 

Wentzel,  K.R.  (1993).  Does  Being  Good  Make  the  Grade?  Social  Behavior  and  Academic  Competence  in 
Middle  School.  Journal  of  Educational  Psychology,  85:  357-364. 

What  Works  Clearinghouse,  (n.d.-a).  Character  Education.  Retrieved  August  24,  2007,  from 

http:/ /ies. ed.gov/ncee/wwc/ reports/Topic.aspx?tid=12. 

What  Works  Clearinghouse,  (n.d.-b).  What  Works  Clearinghouse  Intervention  Hating  Scheme.  U.S.  Department  of 
Education,  National  Center  for  Education  Evaluation  and  Regional  Assistance.  Retrieved  May  6, 
2009,  from  http://ies.ed.gov/ncee/wwc/pdf/rating_scheme.pdf. 

Wheeler,  V.A.,  and  Ladd,  G.W.  (1982).  Assessment  of  Children’s  Self-Efficacy  for  Social  Interactions  With 
Peers.  Developmental  Psychology,  18:  795-805. 

Wilson,  D.B.,  Gottfredson,  D.C.,  and  Najaka,  S.S.  (2001).  School-Based  Prevention  of  Problem  Behavior:  A 
Meta-Analysis.  Journal  of  Quantitative  Criminology,  17:  247-272. 

Wilson,  S.J.,  and  Lipsey,  M.W.  (2007).  School-Based  Interventions  for  Aggressive  and  Disruptive  Behavior: 
Update  of  a Meta-Analysis.  American  Journal  of  Preventive  Medicine,  33{ 2S):  S130-S143. 

Wilson,  S.J.,  Lipsey,  M.W.,  and  Derzon,  J.H.  (2003).  The  Effects  of  School-Based  Intervention  Programs  on 
Aggressive  Behavior:  A Meta- Analysis  .Journal  of  Consulting  and  Clinical  Psychology,  71:  136-149. 

Zellman,  G.L.,  and  Waterman,  J.M.  (1998).  Understanding  the  Impact  of  Parent  School  Involvement  on 
Children’s  Educational  Outcomes.  Journal  of  Educational  Research,  91:  370-380. 


491 


Appendix  A:  Analysis  of  Cohorts  1 and  2 


A-l 


This  page  is  intentionally  blank. 


Analysis  of  Cohorts  1 and  2 


For  Years  2 and  3 of  the  social  and  character  development  (SACD)  multiprogram  evaluation,  additional  funds 
were  provided  to  enable  research  teams  to  add  additional  schools  in  order  to  increase  the  sample  size  and  the 
corresponding  statistical  power  of  the  analyses.  Four  teams  added  2 or  4 schools  and  randomly  assigned  half 
to  the  treatment  and  half  to  the  control  groups  using  the  same  matching  process  as  with  the  original  schools. 
The  additions  were  as  follows:  (1)  University  at  Buffalo,  The  State  University  of  New  York  (. Academic  and 
Behavioral  Competencies  Program  [ABC])  added  2 schools,  (2)  University  of  North  Carolina  at  Chapel  Hill 
( Competence  Support  Program  [CSP])  added  4 schools,1  (3)  Vanderbilt  University  (Love  In  a Big  World  [LBW]) 
added  2 schools,  and  (4)  The  Children’s  Institute  ( Promoting  Alternative  Thinking  Strategies  [PATHS])  added  4 
schools.  As  a result,  a second  set  of  12  schools  (and  a second  cohort  of  students)  was  added  to  the  SACD 
evaluation  for  Years  2 and  3,  raising  the  total  number  of  schools  involved  in  the  study  from  84  to  96  (with  the 
second  cohort  of  schools  making  up  14  percent  of  the  sample).2 

Data  collection  at  these  schools  followed  the  same  process  and  used  the  same  reports  as  at  the  first  set  of 
schools.  Initial  data  collection  with  third-grade  students  in  the  second  cohort  began  in  fall  2005  (1  year  later 
than  the  first  cohort  of  third-graders),  with  the  first  follow-up  at  the  end  of  third  grade  (spring  2006)  and  the 
second  follow-up  at  the  end  of  fourth  grade  (spring  2007).  Students  were  followed  for  2 years  (through  third 
and  fourth  grades)  but  not  through  fifth  grade  as  was  done  with  Cohort  1 . 

This  appendix  examines  whether  an  analysis  of  the  larger  sample  of  both  cohorts  together  (all  Cohort  1 
students  plus  all  Cohort  2 students)  gives  similar  or  different  results  from  those  for  Cohort  1 alone  (presented 
in  chapters  1 through  8).  The  two  cohorts  were  combined  into  one  sample  by  grouping  all  third-graders 
together  and  all  fourth-graders  together.  Two  analyses  were  done.  The  first  compared  student  outcomes  for 
treatment  and  control  students  at  the  end  of  their  fourth-grade  year,  and  the  second  looked  at  average  yearly 
growth  in  student  outcomes  from  the  beginning  of  third  grade  to  the  end  of  fourth  grade.  As  Cohort  2 was 
not  followed  through  the  fifth  grade,  both  analyses  are  based  on  the  third-  and  fourth-grade  data  only. 

Sample 

There  were  7,255  fourth-graders  enrolled  in  one  of  95  study  schools  at  the  time  of  data  collection  in  spring  of 
the  students’  fourth  grade  (spring  2006  for  Cohort  1 and  spring  2007  for  Cohort  2).  These  students  included 
5,750  original  cohort  “stayers”  who  were  enrolled  in  a study  school  at  the  time  of  data  collection  in  fall  of 
their  third  grade  and  spring  of  fourth  grade,  and  1,505  new  entrants  who  enrolled  in  a study  school  after  the 
data  collection  in  the  fall  of  their  third  grade  but  before  data  collection  in  the  spring  of  fourth  grade.  The 
sample  universe  excluded  1 ,607  original  cohort  “leavers”  who  were  enrolled  in  a study  school  in  fall  of  third 
grade  but  not  in  spring  of  fourth  grade. 

There  were  1,064  third-,  fourth-,  or  fifth-grade  teachers  in  spring  of  fourth  grade  but  124  either  did  not  give 
consent  to  participate  or  did  not  provide  the  requested  data.  As  a result,  the  teacher  samples  for  the  spring  of 
fourth-grade  analysis  included  940  third-  to  fifth -grade  teachers  (fourth -grade  teachers  completed  the  Teacher 
Report  on  Student  and  third-  to  fifth-grade  teachers  completed  the  Teacher  Report  on  Classroom  and 
School).  Teachers  who  had  been  in  the  study  in  fall  of  third  grade  but  were  excluded  from  the  fourth-grade 


1 One  of  CSP’s  Cohort  2 schools  stopped  implementing  the  intervention  after  the  first  year  but  remained  in  the  study 
for  the  second  year  as  a treatment  school. 

2 Of  the  original  84  schools,  one  control  school  dropped  out  of  the  study  prior  to  Year  2 when  it  became  a magnet 
school,  leaving  95  schools  available  for  the  analysis  of  both  cohorts  together. 
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follow-up  could  either  have  (1)  left  the  school  between  fall  of  third  grade  and  spring  of  fourth  grade  or  (2) 
remained  at  the  school  but  discontinued  teaching  in  the  third,  fourth,  or  fifth  grade. 

The  consent  and  response  rates  led  to  the  analysis  sample  sizes  and  the  percentages  of  the  universe  of  fourth- 
graders  for  which  there  were  data  (table  A.l).  The  analysis  sample  for  the  Child  Report  was  4,549  (2,395 
treatment  group  students  and  2,154  control  group  students);  the  samples  for  the  Primary  Caregiver  Report 
and  Teacher  Report  on  Student  were  3,551  (1,856  treatment  group  students  and  1,695  control  group 
students)  and  4,737  (2,492  treatment  group  students  and  2,245  control  group  students),  respectively.  These 
Child  Report,  Primary  Caregiver  Report,  and  Teacher  Report  on  Student  samples  reflected  63  percent,  49 
percent,  and  65  percent  of  the  7,255  students  in  the  sample  universe  of  students  in  the  fourth  grade  in  study 
schools  (in  spring  2006  for  Cohort  1 and  spring  2007  for  Cohort  2,  including  both  original  cohort  stayers  and 
new  entrants  in  each  cohort).  The  analysis  sample  for  the  Teacher  Report  on  Classroom  and  School  was  940 
teachers  (482  treatment  group  and  458  control  group),  which  was  88  percent  of  the  universe  of  teachers  in 
the  study  schools.  None  of  the  differences  between  treatment  and  control  groups  in  the  percentage  of  the 
universe  for  which  data  were  collected  was  statistically  significant  when  the  data  were  pooled  across 
programs. 
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Table  A.1.  Combined  Cohorts  1 and  2:  Sample  size  and  percentage  of  sample  universe  for  fourth-graders,  by  program  and  by 
treatment  group  status 

Analysis  sample  size  (Percent  of  sample  universe) 


Teacher  Report  on 

Child  Report Primary  Caregiver  Report  Teacher  Report  on  Student  Classroom  and  School 


Program 

Treatment 

Control 

Total 

Treatment 

Control 

Total 

Treatment 

Control 

Total 

Treatment 

Control 

Total 

All  programs 

2,395 

2,154 

4,549 

1,856 

1,695 

3,551 

2,492 

2,245 

4,737 

482 

458 

940 

(63.9) 

(61.4) 

(62.7) 

(49.5) 

(48.3) 

(48.9) 

(66.5) 

(64.0) 

(65.3) 

(88.6) 

(88.1) 

(88.3) 

ABC 

254 

350 

604 

182 

262 

444 

266 

362 

628 

56 

72 

128 

(59.8)** 

(63.6) 

(61.9) 

(42.8) 

(47.6) 

(45.5) 

(62.6)*** 

(65.8) 

(64.4) 

(77.8) 

(85.7) 

(82.1) 

CSP 

432 

429 

861 

324 

348 

672 

438 

442 

880 

81 

82 

163 

(65.9) 

(60.2) 

(62.9) 

(49.4) 

(48.8) 

(49.1) 

(66.8) 

(62.0) 

(64.3) 

(85.3) 

(85.4) 

(85.3) 

LBW 

357 

287 

644 

314 

234 

548 

385 

299 

684 

77 

55 

132 

(60.9) 

(62.1) 

(61.5) 

(53.6) 

(50.6) 

(52.3) 

(65.7) 

(64.7) 

(65.3) 

(90.6) 

(87.3) 

(89.2) 

PA 

284 

225 

509 

215 

184 

399 

302 

251 

553 

43 

39 

82 

(66.8) 

(66.4) 

(66.6) 

(50.6) 

(54.3) 

(52.2) 

(71.1) 

(74.0) 

(72.4) 

(86.0) 

(86.7) 

(86.3) 

PATHS 

306 

265 

571 

239 

201 

440 

314 

273 

587 

73 

77 

150 

(58.5) 

(52.2) 

(55.4) 

(45.7)** 

(39.6) 

(42.7) 

(60.0) 

(53.7) 

(56.9) 

(85.9) 

(81.1) 

(83.3) 

4Rs 

376 

303 

679 

245 

210 

455 

403 

323 

726 

91 

82 

173 

(62.8) 

(59.4) 

(61.2) 

(40.9) 

(41.2) 

(41.0) 

(67.3) 

(63.3) 

(65.5) 

(96.8) 

(97.6) 

(97.2) 

SS 

386 

295 

681 

337 

256 

593 

384 

295 

679 

61 

51 

112 

(72.4) 

(69.2) 

(71.0) 

(63.2) 

(60.1) 

(61.8) 

(72.0) 

(69.2) 

(70.8) 

(96.8) 

(96.2) 

(96.6) 

**  Treatment  group  significantly  different  from  control  group  at  the  .01  level. 

***  Treatment  group  significantly  different  from  control  group  at  the  .001  level. 
NOTE:  Abbreviations  are 

ABC:  Academic  and  Behavioral  Competencies  Program 
CSP:  Competence  Support  Program 
LBW:  Love  in  a Big  World 
PA:  Positive  Action 

PATHS:  Promoting  Alternative  Thinking  Strategies. 

4Rs:  The  4Rs  Program  (Reading,  Writing,  Respect,  and  Resolution). 

SS:  Second  Step 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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There  were  several  significant  differences  at  the  program  level.  For  the  ABC  program,  there  was  a statistically 
significant  treatment-control  group  difference  in  the  percentages  with  Child  Report  data  (60%  for  the 
treatment  group  and  64%  for  the  control  group)  and  Teacher  Report  on  Student  data  (63%  and  66%, 
respectively).  For  the  PATHS  program,  there  was  a statistically  significant  difference  in  the  percentages  with 
Primary  Caregiver  Report  data  (46%  and  40%,  respectively). 

As  in  the  Cohort  1 study,  the  sample  for  the  evaluation  of  the  two  cohorts  raised  two  cautions  for  the 
interpretation  of  the  results.  First,  because  of  non-consent  and  non-response,  the  impact  estimates  might  not 
generalize  to  the  full  sample  universe  if  the  characteristics  of  students  in  the  study  and  students  not  in  the 
study  differed  in  ways  that  correlated  with  child  outcomes.  Second,  because  the  study  did  not  follow  leavers 
from  the  original  cohort,  the  intent-to-treat  (ITT)  impacts  could  not  be  estimated  using  all  original  cohort 
members.  Instead,  the  evaluation  used  a repeated  cross-sectional  analysis  approach  to  estimate  impacts;  this 
involved  including  all  students  in  the  study  schools  who  had  consent  and  who  provided  data  at  each  follow- 
up point.  These  unbiased  impact  estimates  for  students  in  the  study  reflect  (1)  the  extent  to  which  the 
programs  improved  the  outcomes  of  the  average  student  in  the  study,  and  (2)  treatment-induced  “mobility 
effects”  resulting  from  potential  differences  in  the  average  outcomes  of  treatment  and  control  students  who 
entered  and  left  the  schools  after  random  assignment. 

Initial  Data 

The  examination  of  the  initial  characteristics3  of  students,  families,  teachers,  and  schools  for  the  combined 
cohorts  showed  few  significant  differences  between  the  treatment  and  control  groups  at  the  combined- 
program  level  and  fewer  differences  among  the  programs  than  would  be  expected  by  chance.  These  results 
are  similar  to  those  of  the  Cohort  1 -only  sample.  The  treatment  and  control  groups  in  the  combined  sample 
were  similar  along  all  student  and  parent/ caregiver  characteristics  except  for  household  size  (the  treatment 
group  students  were  more  likely  to  come  from  smaller  households:  4.6  people  versus  4.7  per  household) 
(table  A.2).  There  were  no  significant  differences  between  treatment  and  control  groups  in  the  mean  values  of 
the  outcomes  measuring  children’s  behavior  and  attitudes  (table  A.3).  There  were  no  significant  differences 
between  the  treatment  and  control  groups  in  teachers’  initial  characteristics  (table  A.4),  nor  were  there 
significant  differences  in  the  characteristics  of  treatment  group  schools  and  control  group  schools  (though 
sample  sizes  were  small  for  these  comparisons)  (table  A.5).  There  were  no  differences  between  treatment  and 
control  groups  in  teacher-reported  measures  of  the  school  environment  on  such  dimensions  as  safety, 
participatory  decision  making,  and  work  pressure  (these  data  are  not  shown  in  a table). 


3 For  a list  of  the  characteristics,  see  Initial  Characteristics  in  chapter  1. 
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Table  A.2.  Combined  Cohorts  1 and  2:  Initial  characteristics  of  children,  their  families,  and 
communities 


Characteristic 

Total 

Treatment 

Control 

Student  sample  size 

4,202 

2,202 

2,000 

Student  demographics 

Gender  (percent) 

Male 

47.4 

47.1 

47.7 

Female 

52.6 

52.9 

52.3 

Race/ethnicity  (percent) 

White  (non-Hispanic) 

40.7 

39.0 

42.5 

Black  (non-Hispanic) 

31.9 

33.7 

30.2 

Hispanic 

19.9 

19.8 

20.0 

Other 

7.4 

7.5 

7.3 

Age  (in  years)  (mean) 

8.1 

8.1 

8.1 

Primary  caregiver  and  family  characteristics 

Primary  caregiver’s  age  (in  years)  (mean) 

36.0 

35.9 

36.1 

Primary  caregiver’s  race/ethnicity  (percent) 

White  (non-Hispanic) 

44.6 

42.8 

46.4 

Black  (non-Hispanic) 

31.9 

33.8 

30.0 

Hispanic 

18.2 

18.0 

18.3 

Other 

5.3 

5.4 

5.3 

Primary  caregiver’s  education  (percent) 

Did  not  complete  high  school 

15.5 

14.5 

16.5 

Completed  high  school  or  equivalent 

24.7 

24.2 

25.3 

Some  college 

39.7 

40.7 

38.7 

Bachelor’s  or  higher  degree 

20.0 

20.6 

19.5 

Primary  caregiver’s  employment  (percent) 

Full-time 

47.5 

47.8 

47.3 

Part-time 

14.6 

14.5 

14.7 

Student  and  employed 

4.4 

4.9 

3.9 

Not  employed 

32.4 

31.7 

33.0 

Other 

1.1 

1.1 

1.1 

Primary  caregiver’s  marital  status  (percent) 

Single 

25.0 

25.6 

24.4 

Married 

55.2 

54.5 

56.0 

Separated  or  divorced 

12.7 

12.7 

12.7 

Widowed 

1.4 

1.3 

1.5 

Other1 

5.6 

5.9 

5.4 

Students  who  live  in  one  household  (percent) 

93.6 

93.5 

93.7 

Number  of  individuals  in  household  (mean) 

4.7 

4.6* 

4.7 

See  notes  at  end  of  table. 
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Table  A.2.  Combined  Cohorts  1 and  2:  Initial  characteristics  of  children,  their  families,  and 
communities— Continued 


Characteristic 

Total 

Treatment 

Control 

Primary  caregiver’s  relationship  to  child  (percent) 

Mother  (stepmother) 

85.9 

86.0 

85.8 

Father  (stepfather) 

8.9 

8.5 

9.4 

Grandparent 

3.1 

3.2 

3.0 

Other  relative 

1.3 

1.4 

1.3 

Nonrelative 

0.8 

1.0 

0.6 

Student  lives  with  (percent) 

Mother  (stepmother)  and  father  (stepfather) 

58.4 

57.0 

59.7 

Mother  (stepmother)  only;  father  (stepfather)  not  present 

35.1 

36.3 

33.9 

Father  (stepfather)  only;  mother  (stepmother)  not  present 

2.6 

2.4 

2.8 

Other  relative/nonrelative;  parents  not  present 

4.0 

4.3 

3.6 

Highest  education  of  anyone  in  household  (percent) 

Did  not  complete  high  school 

10.7 

9.6 

11.9 

Completed  high  school  or  equivalent 

22.4 

21.7 

23.1 

Some  college 

40.5 

42.1 

39.0 

Bachelor’s  or  higher  degree 

26.3 

26.6 

26.0 

Total  household  income  (percent) 

Less  than  $20,000 

34.6 

33.7 

35.4 

$20,000  to  $39,999 

24.6 

25.2 

24.1 

$40,000  to  $59,999 

15.0 

15.4 

14.5 

$60,000  or  more 

25.9 

25.7 

26.0 

Household  income-to-poverty  ratio  (percent) 

Below  135  percent 

40.8 

39.3 

42.3 

135  to  185  percent 

18.7 

19.3 

18.1 

Above  1 85  percent 

40.5 

41.4 

39.6 

Alabama  Parenting  Questionnaire — Poor  Monitoring 

and  Supervision  Subscale  (mean) 

1.2 

1.2 

1.2 

Alabama  Parenting  Questionnaire — Positive  Parenting 

Subscale  (mean) 

3.5 

3.5 

3.5 

Confusion,  Hubbub,  and  Order  Scale  (mean) 

2.2 

2.2 

2.2 

Community  characteristics  (mean) 

Community  Risks  Scale 

1.5 

1.5 

1.5 

Community  Resources  Scale 

2.7 

2.7 

2.7 

Child-Centered  Social  Control  Scale 

3.1 

3.0 

3.1 

* Treatment  group  significantly  different  from  control  group  at  the  .05  level. 

1 Categories  combined  to  protect  confidentiality. 

NOTE:  Weights,  which  assign  equal  weight  to  each  school  within  each  of  the  programs  and  to  each  program  across  programs,  were 
used  in  producing  the  treatment,  control,  and  overall  means.  Statistical  tests  were  conducted  using  regressions  that  included  program 
indicators  to  account  for  the  sample  design  and  adjusted  for  clustering  at  the  school  level. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 


A-8 


Appendix  A:  Analysis  of  Cohorts  1 and  2 


Table  A.3.  Combined  Cohorts  1 and  2:  Mean  scores  and  standard  deviations  for  initial  outcome 
measures 


Total  Treatment  Control 


Outcome  measure-Report 

Range 

Mean 

SD 

Mean 

SD 

Mean 

SD 

Social  and  Emotional  Competence  Domain 

Self-Efficacy  for  Peer  Interaction-CR 

1-4 

2.9 

0.6 

2.9 

0.6 

2.9 

0.6 

Normative  Beliefs  About  Aggression-CR 

1-4 

1.2 

0.5 

1.2 

0.5 

1.2 

0.5 

Empathy-CR 

1-3 

2.4 

0.4 

2.4 

0.4 

2.4 

0.4 

Behavior  Domain 

Altruistic  Behavior-CR 

0-3 

1.5 

0.8 

1.4 

0.8 

1.5 

0.8 

Altruistic  Behavior-TRS 

1-4 

1.4 

0.5 

1.4 

0.5 

1.4 

0.4 

Altruistic  Behavior-PCR 

1-4 

2.3 

0.7 

2.3 

0.7 

2.3 

0.8 

Positive  Social  Behavior-TRS 

1-4 

3.0 

0.7 

3.0 

0.7 

3.0 

0.7 

Positive  Social  Behavior-PCR 

1-4 

3.0 

0.5 

3.0 

0.5 

3.0 

0.5 

Problem  Behavior-CR 

0-3 

0.2 

0.4 

0.3 

0.4 

0.2 

0.4 

Problem  Behavior-TRS 

1-4 

1.4 

0.4 

1.4 

0.4 

1.4 

0.4 

Problem  Behavior-PCR 

1-4 

1.6 

0.3 

1.6 

0.3 

1.6 

0.3 

ADHD-Related  Behavior-TRS 

1-4 

1.7 

0.7 

1.7 

0.7 

1.7 

0.6 

Academics  Domain 

Academic  Competence  and  Motivation-TRS 

1-5 

2.9 

0.9 

2.9 

0.9 

2.9 

0.9 

Engagement  with  Leaming-CR 

1-4 

3.7 

0.6 

3.7 

0.7 

3.7 

0.6 

Perceptions  of  School  Climate  Domain 

Positive  School  Orientation-CR 

1-4 

3.2 

0.7 

3.2 

0.7 

3.2 

0.7 

Negative  School  Orientation-CR 

1-4 

1.9 

0.7 

1.9 

0.7 

1.9 

0.7 

Student  Afraid  at  School-CR 

1-4 

2.4 

0.9 

2.4 

0.9 

2.4 

0.9 

Victimization  at  School-CR 

0-3 

0.8 

0.8 

0.8 

0.8 

0.8 

0.8 

Sample  size — PCR 

4,202 

2,202 

2,000 

Sample  size — CR1 

4,486 

2,357 

2,129 

Sample  size — TRS1 

4,578 

2,405 

2,173 

1 Sample  size  may  differ  for  some  outcomes  due  to  nonresponse. 

NOTE:  Abbreviations  are 
CR:  Child  Report 
PCR:  Primary  Caregiver  Report 
TRS:  Teacher  Report  on  Student 
ADHD:  Attention  deficit  hyperactivity  disorder 
SD:  Standard  deviation 

No  findings  were  found  statistically  significant  at  or  below  the  .05  level.  Weights,  which  assign  equal  weight  to  each  school  within 
each  of  the  programs  and  to  each  program  across  programs,  were  used  in  producing  the  treatment,  control,  overall  means,  and 
standard  deviations.  Statistical  tests  were  conducted  using  regressions  that  included  program  indicators  to  account  for  the  sample 
design  and  adjusted  for  clustering  at  the  school  level. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Table  A.4.  Combined  Cohorts  1 and  2:  Initial  characteristics  of  teachers 


Characteristic 

Total 

Treatment 

Control 

Teacher  sample  size 

964 

494 

470 

Gender  (percent) 

Male 

11.8 

12.2 

11.4 

Female 

88.2 

87.8 

88.6 

Race/ethnicity  (percent) 

White  (non-Hispanic) 

76.0 

74.5 

77.4 

Black  (non-Hispanic) 

16.4 

17.6 

15.1 

Hispanic 

4.7 

5.7 

3.7 

Other 

3.0 

2.1 

3.8 

Number  of  years  teaching  (mean) 

12.7 

12.5 

12.9 

Number  of  years  teaching  in  this  school  (mean) 

7.3 

6.9 

7.7 

Type  of  teaching  certificate  (percent) 

Regular  state  certificate  or  advanced  professional  certificate 

84.6 

83.7 

85.6 

Provisional  certificate  (for  those  in  alternative  certification 

programs) 

5.8 

6.7 

4.8 

Probationary  certificate  (for  those  who  have  satisfied  all 
requirements  except  for  completing  the 

probationary  period) 

5.0 

5.5 

4.5 

Emergency  certificate  or  waiver  (for  those  who  must 

complete  a certification  program  to  continue  teaching) 

0.0 

0.0 

0.0 

Other1 

4.6 

4.0 

5.1 

Education 

Less  than  a bachelor’s  degree 

0.0 

0.0 

0.0 

Bachelor’s  degree 

40.3 

39.4 

41.2 

Master’s  degree,  Ph.D. 

56.1 

58.2 

54.0 

Specialist  degree 

2.0 

1.0 

2.9 

Other 

1.7 

1.4 

1.9 

1 Categories,  including  “no  certificate”  and  “temporary  certificate,”  combined  to  protect  confidentiality. 

NOTE:  No  findings  were  found  statistically  significant  at  or  below  the  .05  level.  Weights,  which  assign  equal  weight  to  each  school 
within  each  of  the  programs  and  to  each  program  across  programs,  were  used  in  producing  the  treatment,  control,  and  overall 
means.  Statistical  tests  were  conducted  using  regressions  that  included  program  indicators  to  account  for  the  sample  design  and 
adjusted  for  clustering  at  the  school  level. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Table  A.5.  Combined  Cohorts  1 and  2:  Initial  characteristics  of  schools 


Characteristic 

Total 

Treatment 

Control 

School  sample  size 

96 

48 

48 

Student  race/ethnicity  (percent) 

White  (non-Hispanic) 

36.4 

36.6 

36.1 

Black  (non-Hispanic) 

40.9 

40.4 

41.5 

Hispanic 

18.8 

18.7 

18.9 

Other 

3.9 

4.3 

3.5 

Students  eligible  for  free  or  reduced-price  lunch  (percent) 

63.2 

64.1 

62.4 

Number  of  students  enrolled  (mean) 

552.1 

541.3 

563.3 

Number  of  full-time  teachers  (mean) 

35.7 

34.4 

36.9 

Title  1 status  (percent) 

Title  1 eligible  school 

76.9 

76.1 

77.6 

Schoolwide  Title  1 

68.4 

66.8 

69.9 

Lowest  grade  offered  (percent) 

Prekindergarten 

59.1 

51.4 

67.0 

Kindergarten 

40.9 

48.6 

33.0 

Highest  grade  offered  (percent) 

5th  grade 

54.0 

57.5 

50.5 

6h  grade 

23.5 

20.2 

26.8 

8th  grade 

21.5 

20.3 

22.7 

Location  of  school  (percent) 

City 

57.2 

58.7 

55.7 

Suburbs 

23.7 

22.4 

25.1 

Rural 

19.1 

18.9 

19.2 

Number  of  years  principal  has  been  at  this  school  (mean) 

5.5 

5.0 

5.9 

NOTE:  No  findings  were  found  statistically  significant  at  or  below  the  .05  level.  Weights,  which  assign  equal  weight  to  each  school 
within  each  of  the  programs  and  to  each  program  across  programs,  were  used  in  producing  the  treatment,  control,  and  overall 
means.  Statistical  tests  were  conducted  using  regressions  that  included  program  indicators  to  account  for  the  sample  design  and 
adjusted  for  clustering  at  the  school  level.  Data  are  missing  from  Love  In  a Big  World  for  student  race/ethnicity,  percentage  of 
students  eligible  for  free  or  reduced-price  lunch,  and  number  of  full-time  teachers. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Large  percentages  of  principals  and  teachers  (treatment  and  control)  reported  using  social  and  character 
development  activities  and  receiving  professional  development  in  their  use.  The  percentages  reported  were 
very  similar  to  those  reported  for  Cohort  1,  so  only  a selection  of  indicators  are  shown  in  table  A.6.  Where 
there  were  differences,  they  favored  the  treatment  teachers,  and  all  the  differences  that  appeared  here  also 
appeared  in  the  Cohort  1 -only  data.  These  differences  may  have  reflected  the  fact  that  the  programs  had  been 
implemented  in  most  of  the  study  schools  before  the  teachers  were  surveyed  about  their  use  of  activities. 
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Table  A.6.  Combined  Cohorts  1 and  2:  Teacher  initial  reports  on  use  of  SACD  activities  and 
training  in  SACD  activities 


Teacher  report 

Total 

Treatment 

Control 

Teachers  reporting  on  using  programs  or  activities  in  their 
class  to  promote  the  following  SACD  goals  (percent) 
Violence  prevention  and  peace  promotion 

64.4 

68.2* 

60.7 

Social  and  emotional  development 

72.2 

76.3* 

68.0 

Character  education 

82.8 

84.2 

81.4 

Tolerance  and  diversity 

61.9 

62.4 

61.4 

Risk  prevention  and  health  promotion 

52.0 

50.8 

53.1 

Civic  responsibility  and  community  service 

59.1 

58.2 

60.0 

Behavior  management 

87.3 

89.1 

85.4 

None  of  the  above 

2.4 

1.9 

2.9 

Teachers  using  the  following  materials  in  conjunction  with  social 
and  character  development  activities  (percent) 

Teacher  guides  (manuals,  curricula) 

66.0 

72.0** 

60.1 

Student  materials  (workbooks,  worksheets) 

50.1 

51.4 

48.9 

Instructional  aids  (games,  software,  videos) 

32.6 

36.0* 

29.1 

Giveaways  (bookmarks,  stickers) 

48.0 

45.5 

50.4 

Children’s  literature 

49.7 

52.5 

46.9 

Other  types  of  materials 

12.9 

10.9 

14.9 

Do  not  use  any  of  the  materials  listed  above 

10.1 

9.8 

10.4 

Use  of  instructional  strategies  to  promote  SACD  (percent) 
Teachers  using  any  of  20  instructional  strategies  to  promote 
social  and  character  development  in  the  classroom  (percent) 

99.8 

99.6 

100.0 

Number  of  the  20  strategies  used  by  teachers  to  promote 
social  and  character  development  in  the  classroom  (mean) 

11.7 

12.1* 

11.4 

Professional  development  on  promoting  SACD  (percent) 
Teachers  reporting  participation  in  social  and  character 
development  training  within  the  past  12  month  (percent) 

73.7 

84.7*** 

62.7 

Number  of  hours  of  social  and  character  development 
training  teachers  report  receiving  during  the  past  12 
months  (mean) 

7.6 

8.9** 

6.3 

* Treatment  group  significantly  different  from  control  group  at  the  .05  level. 

**  Treatment  group  significantly  different  from  control  group  at  the  .01  level. 

***  Treatment  group  significantly  different  from  control  group  at  the  .001  level. 

NOTE:  Weights,  which  assign  equal  weight  to  each  school  within  each  of  the  programs  and  to  each  program  across  programs,  were 
used  in  producing  the  treatment,  control,  and  overall  means.  Statistical  tests  were  conducted  using  regressions  that  included 
program  indicators  to  account  for  the  sample  design  and  adjusted  for  clustering  at  the  school  level.  Sample  size  may  differ  for  some 
outcomes  due  to  nonresponse. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Cohort  2 significantly  differed  in  some  ways  from  Cohort  1 when  comparing  students,  teachers,  and  schools 
(these  data  are  not  shown  in  a table),  although  some  of  these  comparisons  were  based  on  small  samples  sizes 
for  Cohort  2.  At  the  combined-program  level,  there  were  no  statistically  significant  differences  between 
Cohort  1 and  Cohort  2 students  in  terms  of  gender,  race/ ethnicity,  or  age.  However,  students  from  Cohort  1 
were  reported  to  come  from  less  disadvantaged  families  than  students  in  Cohort  2.  Cohort  1 parents  or 
primary  caregivers  were  statistically  significantly  more  likely  to  have  a bachelor’s  degree  or  higher,  be 
employed,  be  married,  live  in  a household  that  included  both  a mother  (or  stepmother)  and  a father  (or 
stepfather),  have  a higher  income-to-poverty  threshold  ratio,  and  have  a lower  average  community  risks  scale 
score  (as  reported  by  primary  caregivers).  With  respect  to  the  outcome  measures,  Cohort  1 students  had 
significantly  higher  Academic  Competence  and  Motivation,  higher  Engagement  with  Learning,  fewer 
Normative  Beliefs  About  Aggression,  and  a more  Positive  Orientation  Toward  School,  although  they  had 
lower  Altruistic  Behavior  scores  (Primary  Caregiver  Report).  The  characteristics  of  teachers  were  similar  in 
Cohort  1 and  Cohort  2,  and  there  were  no  statistically  significant  differences  between  the  cohorts  in  teachers’ 
gender,  race/ ethnicity,  years  of  experience,  or  education.  The  Cohort  1 schools  had  a statistically  significantly 
greater  average  number  of  enrolled  students  and  a greater  average  number  of  full-time  teachers  but  there 
were  no  differences  in  the  characteristics  of  the  students  at  the  schools,  such  as  the  distributions  of  students 
across  racial/ ethnic  groups  or  percentages  of  students  eligible  for  free  or  reduced-price  lunches.  Teacher- 
reported  measures  of  the  school  environment  on  such  dimensions  as  safety,  participatory  decision  making, 
and  work  pressure  showed  no  statistically  significant  differences  between  Cohort  1 and  Cohort  2.  There  were 
few  significant  differences  in  the  use  of  SACD  activities  reported  by  the  teachers  in  both  cohorts.  More 
Cohort  1 teachers  reported  using  activities  for  tolerance  and  diversity  (a  difference  of  10  percentile  points). 
More  Cohort  2 teachers  reported  using  character  education  (with  a difference  of  less  than  2 percentile  points) 
and  behavior  management  activities  (a  difference  of  7 percentile  points).  Teachers  reported  no  significant 
differences  in  the  use  of  materials  and  strategies  for  SACD  activities  and  their  participation  in  professional 
development;  however,  there  was  a difference  between  the  cohorts  in  teacher  reports  on  modeling  positive 
social  and  character  traits  with  students;  this  favored  Cohort  1 (87  % versus  81  %). 

The  addition  of  the  Cohort  2 schools  did  not  have  a large  impact  on  the  percentages  of  data  available  for  the 
analysis.  Table  A.7  compares  the  consent  rates,  completion  rates,  and  percentages  of  sample  with  data  for 
each  report  for  Cohort  1 only  versus  Cohorts  1 and  2 combined.  The  rates  for  Cohorts  1 and  2 combined  are 
very  similar  to  those  of  Cohort  1 only.  Significant  differences  between  treatment  and  control  groups  occur,  in 
most  cases,  similarly  in  both.  In  two  cases,  the  combined  cohort  data  show  significant  differences  between 
the  treatment  and  control  groups  (Child  Report  students  with  data  and  students  with  Teacher  Report  on 
Student  data,  both  in  spring  fourth  grade)  that  were  not  found  for  Cohort  1 only.  However,  the  actual 
magnitudes  of  the  differences  are  very  similar  for  both. 
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Table  A.7.  Cohort  1 versus  combined  Cohorts  1 and  2:  Comparison  of  consent  rates,  completion  rates,  and  percentage  of  sample 
with  data 

Cohort  1 Cohorts  1 and  2 Cohort  1 Cohorts  1 and  2 

(Fall  3rd  grade) (Fall  3rd  grade)  (Spring  4th  grade)  (Spring  4th  grade) 

Treat-  Treat-  Treat-  Treat- 


Report 

Total 

ment 

Control 

Total 

ment 

Control 

Total 

ment 

Control 

Total 

ment 

Control 

Student  sample  size 

6,567 

3,367 

3,200 

7,357 

3,764 

3,593 

6,415 

3,327 

3,088 

7,255 

3,747 

3,508 

Child  Report  (percent) 

Primary  caregiver  consent  rate 

65 

67** 

63 

65 

67*** 

63 

67 

67 

66 

66 

67 

64 

Student  completion  rate 

94 

93* 

94 

93 

95 

94 

95 

96 

95 

96 

96 

95 

Students  with  data1 

61 

62* 

60 

61 

63** 

59 

63 

65 

62 

63 

64* 

61 

Primary  Caregiver  Report  (percent) 

Primary  caregiver  consent  rate 

63 

64** 

61 

62 

64*** 

60 

64 

65 

63 

63 

64 

62 

Primary  caregiver  completion  rate 

92 

92 

92 

92 

91 

92 

78 

77 

78 

77 

77 

77 

Primary  caregivers  with  data1 

57 

59** 

56 

57 

59* 

56 

50 

51 

49 

49 

50 

48 

Teacher  Report  on  Student  (percent) 

Primary  caregiver  consent  rate2 

65 

67*** 

63 

65 

67*** 

63 

67 

67 

66 

66 

67 

64 

Teacher  completion  rate 

96 

96 

96 

96 

95 

97 

100 

100 

99 

100 

100 

99 

Students  with  data1 

62 

64** 

61 

62 

64** 

60 

66 

67 

65 

65 

67* 

64 

Teacher  Report  on  Classroom  and  School 
(3rd-  to  5th-grade  teachers)  (percent) 

Teacher  consent  rate 

96 

98*** 

92 

95 

98*** 

93 

95 

97 

94 

95 

95 

94 

Teacher  completion  rate 

91 

90 

93 

91 

90 

93 

94 

94 

94 

93 

93 

93 

Teachers  with  data1 

87 

88 

86 

87 

88 

86 

90 

90 

89 

88 

89 

88 

* Treatment  group  significantly  different  from  control  group  at  the  .05  level. 

**  Treatment  group  significantly  different  from  control  group  at  the  .01  level. 

***  Treatment  group  significantly  different  from  control  group  at  the  .001  level. 

1 Calculated  as  consent  rate  x completion  rate. 

2 The  primary  caregiver  consent  rates  for  the  Child  Report  and  the  Teacher  Report  on  Student  are  identical,  as  the  primary  caregiver  gave  consent  to  both  together. 
SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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The  opportunity  for  the  teams  to  add  the  Cohort  2 schools  to  the  SACD  multiprogram  evaluation  was 
planned  as  a means  to  increase  the  statistical  power  of  the  analyses  by  increasing  the  sample  size.  In  practice, 
Cohort  2 added  12  schools  to  the  sample  of  84  schools.  The  addition  of  new  schools  also  opened  the 
possibility  for  a change  in  the  intraclass  correlations  (the  ICC,  a measure  of  the  percentage  of  the  total 
variance  in  the  outcomes  that  is  between  schools).  An  increase  in  the  sample  size  would  reduce  the  minimum 
detectable  impacts  in  effect  size  units  (MDES)  for  each  outcome  measure;  however,  changes  in  the  ICCs 
could  offset  those  reductions.  In  practice,  the  MDES  for  the  outcome  measures  both  increased  and 
decreased,  with  17  outcomes  having  changes  in  the  thousandths  of  a point  and  3 having  changes  in  the 
hundredths  (the  largest  being  a decline  of  .027).  Table  A.8  compares  the  MDES  for  Cohort  1 versus  the 
combined  Cohorts  1 and  2. 
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Table  A.8.  Comparison  of  Cohort  1 and  Cohorts  1 and  2:  Adjusted  minimum  detectable  effect 
sizes  for  fourth-grade  outcomes  for  combined-program  evaluation 


Outcome  measure-Report 

Cohort  1 

Cohorts  1 and  2 

Social  and  Emotional  Competence  Domain 

Self-Efficacy  for  Peer  Interaction-CR 

0.048 

0.049 

Normative  Beliefs  About  Aggression-CR 

0.081 

0.076 

Empathy-CR 

0.078 

0.071 

Behavior  Domain 

Altruistic  Behavior-CR 

0.057 

0.056 

Altruistic  Behavior-PCR 

0.044 

0.039 

Altruistic  Behavior-TRS 

0.238 

0.211 

Positive  Social  Behavior-PCR 

0.042 

0.048 

Positive  Social  Behavior-TRS 

0.115 

0.113 

Problem  Behavior-CR 

0.086 

0.090 

Problem  Behavior-PCR 

0.040 

0.045 

Problem  Behavior-TRS 

0.086 

0.082 

ADHD-Related  Behavior-TRS 

0.082 

0.083 

Academics  Domain 

Engagement  with  Learning-CR 

0.036 

0.042 

Academic  Competence  and  Motivation-TRS 

0.077 

0.084 

Perceptions  of  School  Climate  Domain 

Positive  School  Orientation-CR 

0.123 

0.113 

Negative  School  Orientation-CR 

0.070 

0.067 

Student  Afraid  at  School-CR 

0.064 

0.063 

Victimization  at  School-CR 

0.063 

0.071 

Feelings  of  Safety-TRCS 

0.179 

0.157 

Student  Support  for  Teachers-TRCS 

0.154 

0.152 

NOTE:  Abbreviations  are 
CR:  Child  Report 
PCR:  Primary  Caregiver  Report 
TRS:  Teacher  Report  on  Student 
TRCS:  Teacher  Report  on  Classroom  and  School 
ADHD:  Attention  deficit  hyperactivity  disorder 
The  minimum  detectable  effect  (MDE)  formula  used  in  the  calculations  is  as  follows: 


MDE  = f actor  (df)  * 


V 


Pi  (-  + -)  + (1  - Pi)  (—  + — ) 
\sT  sr ) \sTnr  srnr) 


where  Sj-and  sc  are  the  number  of  treatment  and  comparison  schools;  nT  and  nc  are  the  average  number  of  students  per 
classroom;  p,  is  the  intraclass  correlation  (ICC)  at  the  school  level;  and  factor(df)  is  a constant  that  depends  on  the  number  of 
degrees  of  freedom  (df)  available  for  analysis  (and  is  2.802  for  the  pooled  analysis).  Estimates  were  adjusted  for  fixed  program 
effects  as  well  as  baseline  covariates. 


SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Analysis 

The  analyses  used  for  the  combined  cohort  data  were  similar  to  those  used  with  the  Cohort  1 data  and 
described  in  chapter  1.  Adjustments  were  made  to  account  for  fewer  data  collection  time  points.  For  example, 
the  analyses  of  the  impacts  on  the  use  of  SACD  activities  in  the  classroom  and  the  school  and  of  the  impact 
on  student  outcomes  used  the  same  analytical  methods  as  did  the  year-by-year  analysis  of  Cohort  1 fourth- 
grade  student  outcomes. 

In  addition,  a similar  growth  curve  analysis  was  done  with  the  combined  cohort  data,  using  student  results 
from  three  time  points  (fall  third  grade,  spring  third  grade,  and  spring  fourth  grade)  rather  than  the  five  time 
points  used  with  the  Cohort  1 growth  analysis.  For  this  reason,  the  combined  cohort  growth  analysis  is  not 
directly  comparable  to  the  Cohort  1 growth  analysis. 

Impacts  on  Use  of  SACD  Activities 

The  analysis  was  based  on  the  Teacher  Report  on  Classroom  and  School  data  collected  when  the  students 
were  in  the  fourth  grade  (school  year  2005-06  for  Cohort  1 and  school  year  2006-07  for  Cohort  2).  In 
addition  to  information  about  the  teachers’  characteristics  and  backgrounds,  this  questionnaire  asked  third-, 
fourth-,  and  fifth-grade  teachers  in  treatment  and  control  group  schools  to  provide  information  about  the 
social  and  character  development  activities  they  used  in  their  classrooms  and  about  the  climate  in  their 
schools. 

The  impacts  of  the  SACD  programs  on  teachers’  reported  use  of  SACD  activities  for  the  combined  cohort 
analysis  were  consistent  with  results  from  the  analyses  of  Cohort  1 data  when  students  were  in  the  fourth 
grade.  Compared  to  control  group  teachers,  treatment  group  teachers  were  more  likely  to  report  engaging  in 
activities  to  promote  social  and  character  development  on  12  out  of  16  outcomes  and  to  report  engaging  in 
similar  activities  linked  to  named  programs  on  14  out  of  14  outcomes  (table  A.  9).  They  also  reported  using 
more  materials  and  strategies  to  promote  social  and  character  development  for  1 3 out  of  29  outcomes  (table 
A.  10)  and  more  participation  in  SACD-related  professional  development  for  2 out  of  9 outcomes  (table 
A.ll).  For  engaging  in  any  SACD  activities,  the  difference  between  treatment  and  control  group  reports  was 
small  but  significant  (95%  versus  91%),  and  for  engaging  in  any  SACD  activities  linked  to  named  programs, 
the  difference  was  larger  (72%  versus  43%).  For  professional  development,  66  percent  of  treatment  teachers 
reported  receiving  SACD  training  over  the  past  12  months  versus  51  percent  of  control  teachers.  Treatment 
teachers  were  not  more  likely  to  report  teacher  and  staff  attitudes  that  might  be  conducive  to  promoting 
social  and  character  development  (these  data  are  not  shown  in  a table). 
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Table  A.9.  Combined  Cohorts  1 and  2:  Impacts  on  classroom  activities 


Classroom  activity 

Treatment 

Control 

Impact 

p-value 

Teacher  sample  size 

481 

459 

Engagement  in  any  activities  to  promote  SACD  goals1  (percent) 
Activities 

Violence  prevention  and  peace  promotion 

76.4* 

66.1 

10.3 

0.002 

Social  and  emotional  development 

84.5* 

61.2 

23.4 

0.000 

Character  education 

91.9* 

77.6 

14.3 

0.000 

Tolerance  and  diversity 

74.9* 

61.9 

13.0 

0.000 

Risk  prevention  and  health  promotion 

63.6 

60.1 

3.6 

0.389 

Civic  responsibility  and  community  service 

60.7 

60.0 

0.7 

0.872 

Any  activity 

95.0* 

90.8 

4.2 

0.025 

Behavior  management 

92.1* 

83.3 

8.8 

0.000 

At  least  1 hour  per  week  of  activities 

Violence  prevention  and  peace  promotion 

41.9* 

25.3 

16.7 

0.000 

Social  and  emotional  development 

49.2* 

25.7 

23.5 

0.000 

Character  education 

59.4* 

31.7 

27.7* 

0.000 

Tolerance  and  diversity 

35.9* 

22.9 

13.0 

0.000 

Risk  prevention  and  health  promotion 

26.9 

22.0 

4.9 

0.174 

Civic  responsibility  and  community  service 

20.8 

16.1 

4.7 

0.111 

Any  activity 

83.7* 

73.2 

10.5 

0.002 

Behavior  management 

74.9* 

61.5 

13.3 

0.002 

Engagement  in  named  activities  to  promote  SACD  goals1  (percent) 
Named  activities 

Violence  prevention  and  peace  promotion 

46.7* 

18.1 

28.6 

0.000 

Social  and  emotional  development 

53.2* 

11.6 

41.6 

0.000 

Character  education 

56.4* 

14.9 

41.5 

0.000 

Tolerance  and  diversity 

36.4* 

8.2 

28.2 

0.000 

Risk  prevention  and  health  promotion 

30.5* 

18.9 

11.5 

0.001 

Civic  responsibility  and  community  service 

14.4* 

5.9 

8.5 

0.000 

Any  named  activity 

71.5* 

42.5 

29.0 

0.000 

At  least  1 hour  per  week  of  named  activities 

Violence  prevention  and  peace  promotion 

31.5* 

11.7 

19.8 

0.000 

Social  and  emotional  development 

36.1* 

7.0 

29.1 

0.000 

Character  education 

41.8* 

8.1 

33.6 

0.000 

Tolerance  and  diversity 

24.2* 

4.2 

20.0 

0.000 

Risk  prevention  and  health  promotion 

17.4* 

10.8 

6.6 

0.035 

Civic  responsibility  and  community  service 

8.1* 

2.2 

6.0 

0.003 

Any  named  activity 

50.7* 

22.6 

28.1 

0.000 

* Treatment  group  significantly  different  from  control  group  at  the  .05  level. 

1 Impact  on  domain  found  statistically  significant  after  adjustments  made  for  multiple  comparisons  within  domain. 

NOTE:  Weights,  which  assign  equal  weight  to  each  school  within  each  of  the  programs  and  to  each  program  across  programs,  were 
used  in  producing  the  treatment,  control,  and  overall  means. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 


A-19 


Appendix  A:  Analysis  of  Cohorts  1 and  2 


Table  A.10.  Combined  Cohorts  1 and  2:  Impacts  on  use  of  SACD  classroom  materials  and 
teaching  strategies 


SACD  material  and  teaching  strategy 

Treatment 

Control 

Impact 

p-value 

Teacher  sample  size 

481 

459 

Use  of  SACD  materials  (percent) 

Teacher  guides  (manuals,  curricula) 

80.4* 

59.5 

20.9 

0.000 

Student  materials  (workbooks  or  sheets) 

66.5* 

54.3 

12.3 

0.001 

Instructional  aids  (games,  software,  videos) 

46.2* 

38.4 

7.8 

0.029 

Giveaways  (bookmarks,  stickers) 

53.1 

49.2 

3.9 

0.250 

Children’s  literature 

57.8 

48.2 

9.6 

0.027 

Other  types  of  materials 

9.1 

12.3 

-3.3 

0.147 

Did  not  use  any  of  these  materials 

4.6* 

13.1 

-8.5 

0.000 

Use  of  teaching  strategies  (percent) 

Role-playing 

88.0* 

67.4 

20.5 

0.000 

Cooperative  learning 

98. 2A 

96.3 

2.0 

0.075 

Peer  group  discussions 

92.4 

89.1 

3.3 

0.111 

Direct  instruction  of  SACD 

93.4* 

78.0 

15.4 

0.000 

Skill  training 

65.3* 

44.2 

21.2 

0.000 

Incorporating  SACD  into  academic  curriculum 

86.3* 

73.3 

13.0 

0.000 

Parent  training 

11.7 

9.7 

2.0 

0.466 

Parent  and  community  involvement 

30.7* 

22.0 

8.7 

0.023 

Mentoring 

46.0 

44.8 

1.1 

0.741 

Good  behavior  notes  sent  home  daily  or  weekly 

79.5 

74.3 

5.2 

0.143 

Presenting  role  models 

75. 2A 

69.6 

5.6 

0.080 

Targeted  story  reading  or  writing  on  social 
and  character  development  themes 

88.5* 

78.9 

9.6 

0.000 

Peer  mediation 

54.4 

52.4 

2.0 

0.635 

Honor  roll  for  positive  behavior 

62.0 

59.9 

2.1 

0.639 

Pledges  or  recitations  on  SACD  themes 

48.0 

44.9 

3.1 

0.598 

Guided  visualization 

60.9* 

48.2 

12.6 

0.002 

Student-led/student-assisted  instruction 

59.6* 

48.5 

11.1 

0.002 

Journaling 

77.9 

72.3 

5.6 

0.117 

Time  out  for  negative  behavior 

86.1 

85.3 

0.8 

0.720 

Daily  or  weekly  rewards  for  positive  behavior 

93.0 

90.2 

2.8 

0.190 

Any  strategy  (percent) 

100.0 

100.0 

0.0 

t 

Number  of  strategies  (mean) 

13.8* 

12.4 

1.5 

0.000 

t Not  applicable. 

* Treatment  group  significantly  different  from  control  group  at  the  .05  level. 

A Treatment  group  significantly  different  from  control  group  at  the  .10  to  > .05  level. 

NOTE:  Weights,  which  assign  equal  weight  to  each  school  within  each  of  the  programs  and  to  each  program  across  programs,  were 
used  in  producing  the  treatment,  control,  and  overall  means. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Table  A.11.  Combined  Cohorts  1 and  2:  Impacts  on  teacher-reported  SACD  professional 
development 


SACD  professional  development1 

Treatment 

Control 

Impact 

p-value 

Teacher  sample  size 

481 

459 

SACD  training  in  past  12  months  (percent) 

65.9* 

51.3 

14.6 

0.000 

Hours  of  SACD  training  (mean) 

5.8 

4.9 

0.9 

0.404 

Training  by  goal  (percent) 

Violence  prevention  and  peace  promotion 

19.1 

14.8 

4.3 

0.180 

Social  and  emotional  development 

23. 0A 

16.6 

6.4 

0.066 

Character  education 

35.1* 

18.4 

16.8 

0.000 

Tolerance  and  diversity 

17.5 

18.9 

-1.4 

0.650 

Risk  prevention  and  health  promotion 

14.2 

14.4 

-0.2 

0.933 

Civic  responsibility  and  community  service 

7.4 

5.3 

2.0 

0.232 

Behavior  management 

29.6 

25.0 

4.6 

0.210 

* Treatment  group  significantly  different  from  control  group  at  the  .05  level. 

A Treatment  group  significantly  different  from  control  group  at  the  .10  to  > .05  level. 

1 Impact  on  domain  found  statistically  significant  after  adjustments  made  for  multiple  comparisons  within  domain  for  all  years. 

NOTE:  Weights,  which  assign  equal  weight  to  each  school  within  each  of  the  programs  and  to  each  program  across  programs,  were 
used  in  producing  the  treatment,  control,  and  overall  means. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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The  impacts  of  the  programs  on  the  SACD  activity  domains  were  consistent  with  results  from  the  analyses  of 
Cohort  1 data  when  students  were  in  the  fourth  grade.  Table  A.  12  shows  that  statistically  significant  larger 
percentages  of  treatment  group  teachers  than  control  group  teachers  reported  engaging  in  activities  to 
promote  social  and  character  development,  using  more  materials  and  strategies  to  promote  social  and 
character  development,  and  participating  in  related  professional  development.  Treatment  group  teachers  were 
not  more  likely  to  report  changes  in  the  school  environment  or  in  teacher  and  staff  attitudes  that  might  be 
conducive  to  promoting  social  and  character  development. 

Table  A.12.  Combined  Cohorts  1 and  2:  Significant  impacts  on  domains  of  use  of  SACD  activity 
overall  and  by  program 


SACD  activity  domain 


Program 

SACD 

activities 

SACD  activities 
linked  to  named 
programs 

Classroom 
materials 
and  strategies 

Schoolwide 

strategies 

Professional 

development 

Attitudes 

and 

practices 

Overall 

1 ,2,3 

1 ,2,3 

1,2,3 

+2'3 

ABC 

+3 

CSP 

+3 

2 

+ 

LBW 

+3 

+3 

2 

+ 

+3 

PA 

+1'3 

PATHS 

+1’3 

+1’3 

4Rs 

+1’3 

+1’3 

SS 

+1’3 

+1'3 

+3 

1 Based  on  univariate  statistical  tests,  at  least  half  of  the  impacts  were  positive  and  statistically  significant  and  no  impact  was 
negative  and  statistically  significant. 

2 The  omnibus  impact  for  all  the  outcomes  measured  together  was  positive  and  statistically  significant  on  the  basis  of  a multivariate 
statistical  test. 

3 At  least  one  outcome  remained  positive  and  statistically  significant  and  no  outcome  was  negative  and  statistically  significant  after 
applying  the  Benjamini-Hochberg  (1 995)  procedure  to  adjust  significance  levels  downward  to  account  for  the  multiple  testing  of 
impacts. 

NOTE:  Abbreviations  are 

ABC:  Academic  and  Behavioral  Competencies  Program 
CSP:  Competence  Support  Program 
LBW:  Love  In  a Big  World 
PA:  Positive  Action 

PATHS:  Promoting  Alternative  Thinking  Strategies 

4Rs:  The  4Rs  Program  (Reading,  Writing,  Respect,  and  Resolution) 

SS:  Second  Step 

+ Statistically  significant  beneficial  impact  on  domain 
Blank  cell:  Finding  of  no  impact 

No  detrimental  impact  was  found  statistically  significant  at  or  below  the  .05  level.  For  a description  of  the  heuristics  used  to 
determine  the  statistically  significant  beneficial  impact  on  the  domain,  see  chapter  1 . 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Impacts  on  Student  Outcomes:  Year-by-Year  and  Growth  Analyses  Results 

Year-by-Year  Analysis 

None  of  the  20  estimated  combined-program  impacts  on  fourth-grade  child  and  school  outcomes,  based  on 
the  combined  samples  of  Cohorts  1 and  2,  were  statistically  significantly  different  from  zero  at  the  5 percent 
level  (table  A.  134).  This  varies  from  the  finding  of  one  statistically  significant  impact  (for  Student  Support  for 
Teachers)  in  the  Cohort  1 analysis.  The  combined-sample  analysis  agreed  with  the  Cohort  1 analysis  that  none 
of  the  impacts  was  substantively  important.  The  estimated  impacts  in  effect  size  units  ranged  from  a 
detrimental  effect  of  -0.06  (for  child-reported  Altruistic  Behavior)  to  a beneficial  effect  of  0.12  standard 
deviations  (for  Student  Support  for  Teachers).  When  the  What  Works  Clearinghouse  heuristics5  were  applied 
to  the  domains  to  adjust  for  multiple  comparisons  within  each  outcome  domain,  the  fourth  heuristic 
identified  the  significance  of  a detrimental  impact  on  children’s  Social  and  Emotional  Competence  (similar  to 
the  finding  in  the  Cohort  1 analysis). 


4 Sample  sizes  are  not  reported  in  the  Results  tables  because  they  vary  by  outcome.  Table  A.20  provides  the  range  of 
sample  sizes  for  the  outcomes  within  each  report. 

5 See  the  chapter  1 section  headed  Year-by-Year  Impacts  on  Students  and  Perceptions  of  School  Climate,  subsection 
Statistical  Significance  and  Substantively  Important  Effects,  for  a discussion  of  these  heuristics. 
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Table  A.13.  Combined  Cohorts  1 and  2:  Combined-program  impacts  on  outcomes  for  fourth- 
graders 


Scale-Report 

Treatment 

Control 

Effect 

size1 

p-value2 

Social  and  Emotional  Competence  Domain3 

Self-Efficacy  for  Peer  Interactions-CR  (+) 

3.17 

3.20 

-0.05 

0.194 

Normative  Beliefs  About  Aggression-CR  (-) 

1.36 

1.37 

-0.01 

0.849 

Empathy-CR  (+) 

2.18 

2.19 

-0.02 

0.699 

Behavior  Domain 

Altruistic  Behavior-CR  (+) 

1.06 

1.10 

-0.06 

0.154 

Altruistic  Behavior-PCR  (+) 

2.27 

2.25 

0.03 

0.364 

Altruistic  Behavior-TRS  (+) 

1.38 

1.36 

0.05 

0.686 

Positive  Social  Behavior-PCR  (+) 

3.06 

3.04 

0.05 

0.177 

Positive  Social  Behavior-TRS  (+) 

3.02 

3.05 

-0.04 

0.533 

Problem  Behavior-CR  (-) 

0.37 

0.36 

0.01 

0.901 

Problem  Behavior-PCR  (-) 

1.54 

1.55 

-0.04 

0.248 

Problem  Behavior-TRS  (-) 

1.45 

1.46 

-0.01 

0.800 

ADHD-Related  Behavior-TRS  (-) 

1.69 

1.70 

-0.02 

0.624 

Academics  Domain 

Engagement  with  Leaming-CR  (+) 

3.66 

3.67 

-0.02 

0.680 

Academic  Competence  and  Motivation-TRS  (+) 

2.97 

2.99 

-0.02 

0.651 

Perceptions  of  School  Climate  Domain 

Positive  School  Orientation-CR  (+) 

2.65 

2.62 

0.04 

0.508 

Negative  School  Orientation-CR  (-) 

1.99 

2.00 

-0.02 

0.653 

Student  Afraid  at  School-CR  (-) 

2.24 

2.28 

-0.04 

0.402 

Victimization  at  School-CR  (-) 

0.68 

0.71 

-0.03 

0.533 

Feelings  of  Safety-TRCS  (+) 

3.42 

3.36 

0.08 

0.325 

Student  Support  for  Teachers-TRCS  (+) 

3.51 

3.41 

0.12 

0.099 

1 The  effect  size  was  calculated  by  dividing  the  estimated  impact  by  the  standard  deviation  of  the  outcome  measure  for  the  control 
group. 

2 The  p-value  is  from  a two-tailed  t test  to  gauge  the  statistical  significance  of  the  impact  estimate. 

3 Impact  on  domain  found  statistically  significant  and  detrimental  based  on  the  fourth  heuristic,  in  which  the  statistical  model  used  to 
estimate  impacts  on  the  individual  outcomes  was  re-estimated  using  a composite  of  all  the  outcome  variables  under  a domain.  The 
domain  was  found  significant  if  the  impact  on  the  composite  was  significant.  The  composite  was  formed  by  standardizing  each 
outcome  variable  using  its  standard  deviation,  combining  the  values  of  the  outcome  variables,  and  taking  the  average  of  the  final 
value. 

NOTE:  Abbreviations  are 
CR:  Child  Report 
PCR:  Primary  Caregiver  Report 
TRS:  Teacher  Report  on  Student 
TRCS:  Teacher  Report  on  Classroom  and  School 
ADHD:  Attention  deficit  hyperactivity  disorder 

The  +/-  signs  in  parentheses  indicate  the  direction  of  a beneficial  outcome.  No  findings  were  found  statistically  significant  at  or  below 
the  .05  level.  All  impact  estimates  were  calculated  using  regression  models  in  which  each  program  and  school  within  a program  was 
weighted  equally.  The  standard  errors  of  all  estimates  account  for  design  effects  due  to  unequal  weighting  and  the  clustering  of 
students  within  schools. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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The  lack  of  statistically  significant  beneficial  impact  estimates  at  the  combined -program  level  might  have  been 
due  to  beneficial  impacts  in  some  programs  that  were  offset  by  negative  impacts  in  others.  Differences  in 
estimated  impacts  across  programs  were  tested  for  as  were  the  statistical  significance  of  program-specific 
impact  estimates. 

With  two  exceptions,  the  20  impact  estimates  did  not  significantly  differ  by  program.  For  18  of  the  20 
outcomes,  the  differences  in  pooled  impact  estimates  across  programs  were  not  statistically  significant  at  the  5 
percent  level.  For  these  outcomes,  one  or  two  programs  were  not  driving  or  masking  the  estimated  impacts  in 
the  combined  program  data.  For  the  other  two  outcomes  (Academic  Competence  and  Motivation,  and 
Feelings  of  Safety)  the  ABC  program  showed  a beneficial,  statistically  significant  result  so  the  hypothesis  that 
the  estimated  impacts  were  equal  across  programs  was  rejected. 

From  the  analyses  of  the  individual  SACD  programs,  6 of  140  program  impact  estimates  were  statistically 
significant  (versus  7 that  would  be  expected  by  chance  and  the  7 that  were  found  in  the  fourth-grade  analysis 
of  Cohort  1).  Four  of  these  were  beneficial  and  2 were  detrimental  (versus  the  5 beneficial  and  2 detrimental 
found  in  the  analysis  of  Cohort  1).  For  the  three  programs  that  did  not  add  a second  cohort  (Positive  Action 
[PA],  the  4Rs  Program  [4Rs],  and  Second  Step  [SS]),  results  remained  the  same  as  in  the  Cohort  1 analysis  (2 
significant  beneficial  impacts,  1 significant  detrimental  impact,  and  2 non-significant  but  substantively 
important  beneficial  impacts:  see  table  1.27  in  chapter  1).  For  the  four  programs  that  included  a second 
cohort,  table  A.  14  identifies  the  statistically  significant  and  the  non-significant  substantively  important  results 
from  the  combined  cohort  analysis  and  compares  them  with  the  similar  results  found  in  the  Cohort  1 only 
analysis  (table  A.  15  compares  all  the  results). 

With  respect  to  the  ABC  program,  statistically  significant  beneficial  impacts  were  found  on  Academic 
Competence  and  Motivation  (effect  size  of  0.35)  and  on  Feelings  of  Safety  (effect  size  of  0.67),  and  these 
align  with  findings  from  the  Cohort  1 analysis.  However,  the  combined  cohort  analysis  did  not  find  a 
significant  detrimental  impact  on  Altruistic  Behavior  (Child  Report),  which  did  appear  in  the  Cohort  1 
analysis.  With  respect  to  CSP,  the  combined  cohort  findings  showed  a statistically  significant  detrimental 
impact  on  Empathy  (effect  size  of  -0.18),  which  did  not  occur  in  the  Cohort  1 analysis.  CSP’s  Cohort  1 
beneficial  impact  on  Problem  Behavior  disappeared  in  the  combined  cohort  analysis.  LBW  and  PATHS  had 
no  significant  impacts  in  either  the  Cohort  1 or  the  combined  cohort  analyses. 

With  respect  to  non-significant  but  substantively  important  results,  when  the  combined  cohort  analyses  were 
compared  to  the  Cohort  1 analyses,  ABC  maintained  its  beneficial  impact  for  Student  Support  for  Teachers 
(effect  size  of  .40).  CSP  lost  its  beneficial  effect  for  Altruistic  Behavior  and  Student  Afraid  at  School.  LBW 
lost  its  beneficial  impact  for  Student  Support  for  Teachers  and  its  detrimental  effect  on  Altruistic  Behavior 
(Teacher  Report  on  Student),  and  it  gained  a detrimental  effect  on  Positive  Social  Behavior  (Teacher  Report 
on  Student)  (effect  size  of  -.25).  PATHS  gained  a beneficial  impact  on  Altruistic  Behavior  (Teacher  Report  on 
Student). 
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Table  A.14.  Comparison  of  individual  programs’  statistically  significant  impacts  and  nonsignificant  substantive  impacts  between 
Cohort  1 and  combined  Cohort  1 and  Cohort  2 fourth-graders 

Statistically  significant1 Nonstatistically  significant  but  substantive2 

Beneficial  impacts  Detrimental  impacts  Beneficial  impacts  Detrimental  impacts 


Program3 (Report)  (Effect  size)  (p-value)  (Report)  (Effect  size)  (p-value)  (Report)  (Effect  size)  (p-value)  (Report)  (Effect  size)  (p-value) 


Total 
Cohort  1 
Cohorts 
1 and  2 

3 

1 

4 

1 

2 

1 

2 

1 

ABC 
Cohort  1 

Cohorts 
1 and  2 

Academic  Competence 
(CR)  (.31)  (.011) 
Feelings  of  Safety 
(TRCS)  (.75)  (.003) 

Altruistic  Behavior 
(CR)  (-.20)  (.029) 

Student  Support  for  Teachers 
(TRCS)  (.27)  (.276) 

Academic  Competence 
(CR)  (.35)  (.002) 
Feelings  of  Safety 
(TRCS)  (.67)  (.001) 

Student  Support  for  Teachers 
(TRCS)  (.40)  (.105) 

CSP 
Cohort  1 

Cohorts 
1 and  2 

Problem  Behavior 
(PCR)  (-.21)  (.042) 

Altruistic  Behavior 
(TRS)  (.47)  (.132) 
Student  Afraid  at  School 
(CR)  (-.26)  (.090) 

Empathy 
(CR)  (-.18)  (.041) 

LBW 
Cohort  1 

Cohorts 
1 and  2 

Student  Support  for  Teachers 
(TRCS)  (.28)  (.428) 

Altruistic  Behavior 
(TRS)  (-.34)  (.270) 

Positive  Social  Behavior 
(TRS)  (-.25)  (.205) 

See  notes  at  end  of  table. 
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Table  A. 14.  Comparison  of  individual  programs’  statistically  significant  impacts  and  nonsignificant  substantive  impacts  between 
Cohort  1 and  combined  Cohort  1 and  Cohort  2 fourth-graders — Continued 


Program3 

Statistically  significant1 

Nonstatistically  significant  but  substantive2 

Beneficial  impacts 
(Report)  (Effect  size)  (p-value) 

Detrimental  impacts 
(Report)  (Effect  size)  (p-value) 

Beneficial  impacts 
(Report)  (Effect  size)  (p-value) 

Detrimental  impacts 
(Report)  (Effect  size)  (p-value) 

PATHS 

Cohort  1 

Cohorts 

Altruistic  Behavior 

1 and  2 

(TRS)  (.34)  (.250) 

1 Of  the  80  comparisons  (4  programs  with  20  outcomes  apiece)  made,  4 would  be  expected  to  be  statistically  significant  at  the  .05  level  by  chance. 

2 Defined  as  impacts  that  were  not  statistically  significant  but  were  .25  standard  deviation  units  (absolute  value)  or  more  in  magnitude. 

3 Only  the  four  programs  with  a Cohort  2 are  shown.  The  results  for  the  other  programs  remained  the  same  as  those  discussed  in  chapter  1 . 


NOTE:  Abbreviations  are 

ABC:  Academic  and  Behavioral  Competencies  Program 

CSP:  Competence  Support  Program 

LBW:  Love  In  a Big  World 

PATHS:  Promoting  Alternative  Thinking  Strategies 

CR:  Child  Report 

TRS:  Teacher  Report  on  Student 

PCR:  Primary  Caregiver  Report 

TRCS:  Teacher  Report  on  Classroom  and  School 

Blank  cell:  Finding  of  no  impact 

All  impact  estimates  were  calculated  using  regression  models  in  which  each  school  within  a program  was  weighted  equally.  The  standard  errors  of  all  estimates  account  for  design 
effects  due  to  unequal  weighting  and  the  clustering  of  students  within  schools. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Table  A.15.  Cohort  1 and  combined  Cohorts  1 and  2:  Comparison  of  program  effects 

Panel  1 : ABC  Program 

Cohort  1 Combined  Cohorts  1 and  2 

(Spring  4th  grade) (Spring  4th  grade) 


Treat-  Effect  Treat-  Effect 


Scale-Report 

ment 

Control 

size 

p-value 

ment 

Control 

size 

p-value 

Social  and  Emotional  Competence  Domain 

Self-Efficacy  for  Peer  Interactions-CR  (+) 

3.31 

3.29 

0.03 

0.770 

3.27 

3.24 

0.04 

0.697 

Normative  Beliefs  About  Aggression-CR  (-) 

1.39 

1.29 

0.18 

0.141 

1.38A 

1.28 

0.18 

0.082 

Empathy-CR  (+) 

2.08 

2.19 

-0.21 

0.144 

2.09 

2.17 

-0.17 

0.156 

Behavior  Domain 

Altruistic  Behavior-CR  (+) 

0.94* 

1.09 

-0.20 

0.029 

1.00 

1.13 

-0.16 

0.121 

Altruistic  Behavior-PCR  (+) 

2.14 

2.19 

-0.06 

0.595 

2.19 

2.23 

-0.04 

0.710 

Altruistic  Behavior-TRS  (+) 

1.24 

1.26 

-0.03 

0.914 

1.24 

1.25 

-0.03 

0.920 

Positive  Social  Behavior-PCR  (+) 

3.06A 

2.96 

0.16 

0.071 

3.01 

2.90 

0.18 

0.187 

Positive  Social  Behavior-TRS  (+) 

3.01 

3.02 

-0.01 

0.961 

3.05 

2.95 

0.13 

0.504 

Problem  Behavior-CR  (-) 

0.33 

0.36 

-0.05 

0.659 

0.36 

0.42 

-0.10 

0.545 

Problem  Behavior-PCR  (-) 

1.56 

1.61 

-0.11 

0.229 

1.58 

1.65 

-0.16 

0.198 

Problem  Behavior-TRS  (-) 

1.43 

1.46 

-0.07 

0.655 

1.44 

1.49 

-0.11 

0.403 

ADHD-Related  Behavior-TRS  (-) 

1.69 

1.77 

-0.13 

0.525 

1.68 

1.77 

-0.16 

0.341 

Academics  Domain 

Engagement  with  Learning-CR  (+) 

3.68 

3.63 

0.10 

0.281 

3.69A 

3.60 

0.14 

0.093 

Academic  Competence  and  Motivation-TRS  (+) 

2.92* 

2.65 

0.31 

0.011 

2.93* 

2.63 

0.35 

0.002 

Perceptions  of  School  Climate  Domain 

Positive  School  Orientation-CR  (+) 

2.67 

2.61 

0.07 

0.668 

2.62 

2.56 

0.08 

0.639 

Negative  School  Orientation-CR  (-) 

1.97 

1.96 

0.01 

0.963 

2.01 

2.01 

-0.01 

0.960 

Student  Afraid  at  School-CR  (-) 

2.21 

2.36 

-0.17 

0.263 

2.23 

2.39 

-0.17 

0.211 

Victimization  at  School-CR  (-) 

0.58 

0.66 

-0.11 

0.213 

0.71 

0.70 

0.02 

0.923 

Feelings  of  Safety-TRCS  (+) 

3.59* 

2.97 

0.75 

0.003 

3.51* 

2.94 

0.67 

0.001 

Student  Support  for  Teachers-TRCS  (+) 

3.48 

3.26 

0.27° 

0.276 

3.49 

3.15 

0.40° 

0.105 

See  notes  at  end  of  table. 
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Table  A.15.  Cohort  1 and  combined  Cohorts  1 and  2:  Comparison  of  program  effects — Continued 

Panel  2:  CSP 

Cohort  1 Combined  Cohorts  1 and  2 

(Spring  4th  grade) (Spring  4th  grade) 


Treat-  Effect  Treat-  Effect 


Scale-Report 

ment 

Control 

size 

p-value 

ment 

Control 

size 

p-value 

Social  and  Emotional  Competence  Domain 

Self-Efficacy  for  Peer  Interactions-CR  (+) 

3.15 

3.15 

0.00 

0.978 

3.16 

3.13 

0.05 

0.475 

Normative  Beliefs  About  Aggression-CR  (-) 

1.35 

1.29 

0.12 

0.273 

1.42 

1.29 

0.24 

0.133 

Empathy-CR  (+) 

2.18 

2.23 

-0.11 

0.206 

2.15* 

2.23 

-0.18 

0.041 

Behavior  Domain 

Altruistic  Behavior-CR  (+) 

1.06 

1.00 

0.08 

0.505 

1.01 

1.00 

0.01 

0.889 

Altruistic  Behavior-PCR  (+) 

2.33 

2.31 

0.03 

0.849 

2.34 

2.28 

0.08 

0.452 

Altruistic  Behavior-TRS  (+) 

1.39 

1.25 

0.47° 

0.132 

1.40 

1.30 

0.24 

0.263 

Positive  Social  Behavior-PCR  (+) 

3.11 

3.02 

0.17 

0.125 

3.11 

3.02 

0.15 

0.153 

Positive  Social  Behavior-TRS  (+) 

3.02 

2.93 

0.12 

0.426 

3.02 

3.05 

-0.05 

0.702 

Problem  Behavior-CR  (-) 

0.32 

0.26 

0.15 

0.428 

0.36 

0.27 

0.21 

0.254 

Problem  Behavior-PCR  (-) 

1.50* 

1.56 

-0.21 

0.042 

1.50A 

1.56 

-0.18 

0.069 

Problem  Behavior-TRS  (-) 

1.48 

1.50 

-0.03 

0.852 

1.46 

1.44 

0.04 

0.784 

ADHD-Related  Behavior-TRS  (-) 

1.70 

1.74 

-0.06 

0.585 

1.70 

1.68 

0.04 

0.733 

Academics  Domain 

Engagement  with  Learning-CR  (+) 

3.71 

3.71 

-0.01 

0.952 

3.67 

3.68 

-0.02 

0.889 

Academic  Competence  and  Motivation-TRS  (+) 

2.96 

2.93 

0.04 

0.622 

3.04 

3.09 

-0.05 

0.484 

Perceptions  of  School  Climate  Domain 

Positive  School  Orientation-CR  (+) 

2.64 

2.65 

-0.02 

0.922 

2.56 

2.58 

-0.02 

0.898 

Negative  School  Orientation-CR  (-) 

1.95 

1.98 

-0.05 

0.699 

2.02 

2.00 

0.04 

0.794 

Student  Afraid  at  School-CR  (-) 

2.17A 

2.38 

-0.26° 

0.090 

2.26 

2.31 

-0.06 

0.697 

Victimization  at  School-CR  (-) 

0.69 

0.70 

-0.01 

0.947 

0.69 

0.71 

-0.03 

0.756 

Feelings  of  Safety-TRCS  (+) 

3.60 

3.64 

-0.05 

0.834 

3.63 

3.65 

-0.02 

0.919 

Student  Support  for  Teachers-TRCS  (+) 

3.55 

3.48 

0.10 

0.551 

3.59 

3.55 

0.06 

0.634 

See  notes  at  end  of  table. 
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Table  A.15.  Cohort  1 and  combined  Cohorts  1 and  2:  Comparison  of  program  effects — Continued 

Panel  3:  LBW 

Cohort  1 Combined  Cohorts  1 and  2 

(Spring  4th  grade) (Spring  4th  grade) 


Treat-  Effect  Treat-  Effect 


Scale-Report 

ment 

Control 

size 

p-value 

ment 

Control 

size 

p-value 

Social  and  Emotional  Competence  Domain 

Self-Efficacy  for  Peer  Interactions-CR  (+) 

3.19 

3.21 

-0.04 

0.674 

3.20 

3.17 

0.05 

0.602 

Normative  Beliefs  About  Aggression-CR  (-) 

1.26 

1.21 

0.12 

0.368 

1.28 

1.21 

0.19 

0.124 

Empathy-CR  (+) 

2.30 

2.30 

-0.01 

0.963 

2.30 

2.31 

-0.03 

0.752 

Behavior  Domain 

Altruistic  Behavior-CR  (+) 

1.02 

1.16 

-0.20 

0.191 

1.00 

1.13 

-0.18 

0.192 

Altruistic  Behavior-PCR  (+) 

2.22 

2.11 

0.17 

0.152 

2.26A 

2.11 

0.23 

0.092 

Altruistic  Behavior-TRS  (+) 

1.29 

1.46 

-0.34° 

0.270 

1.34 

1.41 

-0.16 

0.607 

Positive  Social  Behavior-PCR  (+) 

3.12 

3.04 

0.13 

0.250 

3.10 

3.03 

0.12 

0.262 

Positive  Social  Behavior-TRS  (+) 

2.94 

3.11 

-0.23 

0.183 

2.95 

3.13 

-0.25° 

0.205 

Problem  Behavior-CR  (-) 

0.25 

0.23 

0.04 

0.728 

0.26 

0.25 

0.02 

0.854 

Problem  Behavior-PCR  (-) 

1.56 

1.56 

-0.01 

0.898 

1.56 

1.57 

-0.03 

0.796 

Problem  Behavior-TRS  (-) 

1.46 

1.40 

0.13 

0.117 

1.49A 

1.39 

0.22 

0.063 

ADHD-Related  Behavior-TRS  (-) 

1.73 

1.64 

0.13 

0.105 

1.78A 

1.68 

0.16 

0.077 

Academics  Domain 

Engagement  with  Learning-CR  (+) 

3.67 

3.69 

-0.04 

0.652 

3.64 

3.68 

-0.08 

0.370 

Academic  Competence  and  Motivation-TRS  (+) 

3.05 

2.99 

0.07 

0.622 

3.09 

2.94 

0.15 

0.378 

Perceptions  of  School  Climate  Domain 

Positive  School  Orientation-CR  (+) 

2.77 

2.79 

-0.03 

0.794 

2.77 

2.82 

-0.07 

0.551 

Negative  School  Orientation-CR  (-) 

1.87 

1.89 

-0.03 

0.846 

1.90 

1.90 

-0.01 

0.960 

Student  Afraid  at  School-CR  (-) 

2.27 

2.23 

0.06 

0.777 

2.27 

2.22 

0.06 

0.727 

Victimization  at  School-CR  (-) 

0.70 

0.71 

-0.01 

0.929 

0.69 

0.70 

-0.02 

0.828 

Feelings  of  Safety-TRCS  (+) 

3.39 

3.45 

-0.06 

0.825 

3.35 

3.53 

-0.24 

0.349 

Student  Support  for  Teachers-TRCS  (+) 

3.57 

3.40 

0.28° 

0.428 

3.53 

3.41 

0.20 

0.494 

See  notes  at  end  of  table. 
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Table  A.15.  Cohort  1 and  combined  Cohorts  1 and  2:  Comparison  of  program  effects — Continued 

Panel  4:  PATHS 

Cohort  1 Combined  Cohorts  1 and  2 

(Spring  4th  grade) (Spring  4th  grade) 


Treat-  Effect  Treat-  Effect 


Scale-Report 

ment 

Control 

size 

p-value 

ment 

Control 

size 

p-value 

Social  and  Emotional  Competence  Domain 

Self-Efficacy  for  Peer  Interactions-CR  (+) 

3.18 

3.28 

-0.16 

0.261 

3.1 2A 

3.26 

-0.21 

0.084 

Normative  Beliefs  About  Aggression-CR  (-) 

1.26 

1.37 

-0.17 

0.214 

1.28 

1.37 

-0.14 

0.251 

Empathy-CR  (+) 

2.21 

2.14 

0.13 

0.245 

2.23 

2.16 

0.13 

0.317 

Behavior  Domain 

Altruistic  Behavior-CR  (+) 

1.00 

0.96 

0.06 

0.635 

1.10 

1.03 

0.08 

0.360 

Altruistic  Behavior-PCR  (+) 

2.06 

2.19 

-0.16 

0.140 

2.18 

2.21 

-0.04 

0.756 

Altruistic  Behavior-TRS  (+) 

1.22 

1.18 

0.17 

0.395 

1.35 

1.24 

0.34° 

0.250 

Positive  Social  Behavior-PCR  (+) 

3.05 

3.06 

-0.01 

0.953 

3.03 

3.03 

-0.01 

0.878 

Positive  Social  Behavior-TRS  (+) 

3.14 

3.04 

0.15 

0.503 

3.06 

3.06 

0.00 

0.997 

Problem  Behavior-CR  (-) 

0.27 

0.30 

-0.06 

0.547 

0.34 

0.31 

0.06 

0.656 

Problem  Behavior-PCR  (-) 

1.53 

1.53 

0.00 

0.987 

1.54 

1.54 

0.02 

0.869 

Problem  Behavior-TRS  (-) 

1.42 

1.42 

0.01 

0.957 

1.46 

1.47 

-0.01 

0.935 

ADHD-Related  Behavior-TRS  (-) 

1.62 

1.70 

-0.13 

0.574 

1.65 

1.73 

-0.14 

0.571 

Academics  Domain 

Engagement  with  Learning-CR  (+) 

3.70 

3.69 

0.01 

0.954 

3.68 

3.69 

-0.03 

0.758 

Academic  Competence  and  Motivation-TRS  (+) 

2.95 

3.05 

-0.09 

0.272 

2.85A 

3.03 

-0.21 

0.093 

See  notes  at  end  of  table. 
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Table  A.15.  Cohort  1 and  combined  Cohorts  1 and  2:  Comparison  of  program  effects — Continued 

Panel  4:  PATHS 

Cohort  1 Combined  Cohorts  1 and  2 

(Spring  4th  grade) (Spring  4th  grade) 


Treat-  Effect  Treat-  Effect 


Scale-Report 

ment 

Control 

size 

p-value 

ment 

Control 

size 

p-value 

Perceptions  of  School  Climate  Domain 
Positive  School  Orientation-CR  (+) 

2.79 

2.67 

0.15 

0.640 

2.81 

2.67 

0.18 

0.461 

Negative  School  Orientation-CR  (-) 

1.87 

1.90 

-0.05 

0.765 

1.95 

1.93 

0.02 

0.825 

Student  Afraid  at  School-CR  (-) 

2.08 

2.09 

-0.01 

0.909 

2.22 

2.20 

0.02 

0.844 

Victimization  at  School-CR  (-) 

0.65 

0.56 

0.12 

0.306 

0.69 

0.63 

0.08 

0.382 

Feelings  of  Safety-TRCS  (+) 

3.36 

3.18 

0.20 

0.615 

3.09 

3.11 

-0.01 

0.962 

Student  Support  for  Teachers-TRCS  (+) 

3.52 

3.46 

0.07 

0.838 

3.28 

3.37 

-0.11 

0.719 

* Treatment  group  significantly  different  from  control  group  at  the  .05  level. 

A Treatment  group  significantly  different  from  control  group  at  the  .10  to  > .05  level. 
s Substantive  (but  nonsignificant  at  .05  level)  effect  size  of  > .25  or  < -.25. 

NOTE:  Abbreviations  are 

ABC:  Academic  and  Behavioral  Competencies  Program 

CSP:  Competence  Support  Program 

LBW:  Love  In  a Big  World 

PATHS:  Promoting  Alternative  Thinking  Strategies 

CR:  Child  Report 

PCR:  Primary  Caregiver  Report 

TRS:  Teacher  Report  on  Student 

TRCS:  Teacher  Report  on  Classroom  and  School 

ADHD:  Attention  deficit  hyperactivity  disorder 

The  +/-  signs  in  parentheses  indicate  the  direction  of  a beneficial  outcome.  All  impact  estimates  were  calculated  using  regression  models  in  which  each  program  and  school  within  a 
program  was  weighted  equally.  The  standard  errors  of  all  estimates  account  for  design  effects  due  to  unequal  weighting  and  the  clustering  of  students  within  schools.  The  effect  size 
was  calculated  by  dividing  the  estimated  impact  by  the  standard  deviation  of  the  outcome  measure  for  the  control  group. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Eight  subgroup  analyses  were  done  (as  discussed  in  chapter  1)  to  determine  whether  the  SACD  programs 
might  be  more  effective  for  some  groups  of  children  than  for  others.  These  subgroups  were  defined  by 
gender,  stayers  versus  new  entrants,  five  types  of  risk  faced  by  the  children,  and  fidelity  of  implementation. 
The  combined  cohort  subgroup  analyses  found  a total  of  20  significant  impacts  compared  to  the  22  found  in 
the  Cohort  1 subgroup  analyses.  The  combined  cohort  analysis  found  14  of  the  same  significant  impacts  as 
the  Cohort  1 analysis,  failed  to  find  8 of  the  Cohort  1 impacts,  and  found  6 different  significant  impacts  that 
did  not  appear  in  the  Cohort  1 analyses.  Of  the  8 results  that  were  significant  in  the  Cohort  1 analyses  but  not 
in  the  combined  cohort  analyses,  7 were  related  to  risk  subgroups  (5  beneficial  impacts  for  high-risk  students 
and  2 detrimental  impacts  for  high-risk  students),  and  1 impact  was  beneficial  for  girls.  Of  the  6 new  findings 
in  the  combined  cohort  analyses,  4 occurred  in  the  stayer  versus  new  entrant  subgroup  (2  beneficial  impacts 
and  2 detrimental  impacts  for  new  entrants),  1 was  a detrimental  impact  for  high-risk  students,  and  1 was  a 
beneficial  impact  for  high-fidelity  schools. 

Overall,  the  differences  between  the  subgroup  findings  from  the  combined  cohort  analyses  versus  the  Cohort 
1 analysis  are  primarily  less  evidence  that  impacts  are  more  beneficial  for  higher  risk  students  and  mixed 
evidence  on  the  impacts  for  stayers  versus  new  entrants.  Table  A.  16  compares  the  significant  subgroup 
findings  from  the  combined  cohort  analysis  to  those  of  the  Cohort  1 analysis,  showing  each  subgroup  on  a 
separate  panel  (exception:  no  panel  displays  the  socioeconomic  risk  subgroup  analysis  as  no  significant  results 
were  found  in  either  the  Cohort  1 or  the  combined  cohort  analyses). 
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Table  A.16.  Cohort  1 versus  Cohorts  1 and  2:  Comparison  of  significant  impacts  by  subgroup 


Panel  1 : Gender 


Scale-Report1 

Cohort  1 

Cohorts  1 and  2 

Boys 

Girls 

p-value  for 

Boys 

Girls 

p-value  for 

Effect 

size 

p-value 

Effect 

size 

p-value 

test  of 
differences 

Effect 

size 

p-value 

Effect 

size 

p-value 

test  of 
differences 

Normative  Beliefs  About  Aggression-CR  (-) 

0.04 

0.574 

-0.1 0A 

0.095 

0.023* 

0.05 

0.436 

-0.08 

0.165 

0.022* 

Altruistic  Behavior-CR  (+) 

-0.13* 

0.020 

-0.00 

0.967 

0.022* 

-0.1 0A 

0.080 

-0.04 

0.491 

0.293 

Positive  Social  Behavior-PCR  (+) 

0.10* 

0.032 

-0.02 

0.653 

0.033* 

0.10* 

0.036 

-0.02 

0.658 

0.021* 

Engagement  with  Learning-CR  (+) 

-0.10* 

0.050 

0.05 

0.297 

0.011* 

-0.07 

0.183 

0.05 

0.306 

0.035* 

Negative  School  Orientation-CR  (-) 

0.09 

0.126 

-0.14* 

0.011 

0.000* 

0.06 

0.303 

-0.1 0A 

0.083 

0.001** 

See  notes  at  end  of  table. 

Panel  2:  Stayers  and  new  entrants 

Cohort  1 

Cohorts  1 and  2 

Stayers 

Entrants 

p-value  for 

Stayers 

Entrants 

p-value  for 

Effect 

Effect 

test  of 

Effect 

Effect 

test  of 

Scale-Report1 

size 

p-value 

size 

p-value 

differences 

size 

p-value 

size 

p-value 

differences 

Altruistic  Behavior-CR  (+) 

-0.03 

0.516 

-0.1 4A 

0.066 

0.132 

-0.03 

0.565 

-0.1 4A 

0.055 

0.041* 

Positive  Social  Behavior-PCR  (+) 

0.01 

0.813 

0.13A 

0.093 

0.067A 

0.00 

0.970 

0.13* 

0.050 

0.009** 

Problem  Behavior-PCR  (-) 

-0.05 

0.244 

-0.11 

0.160 

0.195 

-0.00 

0.993 

-0.1 4A 

0.074 

0.005** 

Negative  School  Orientation-CR  (-) 

-0.05 

0.432 

0.08 

0.280 

0.053A 

-0.06 

0.297 

0.06 

0.467 

0.047* 

See  notes  at  end  of  table. 
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Table  A.16.  Cohort  1 versus  Cohorts  1 and  2:  Comparison  of  significant  impacts  by  subgroup — Continued 


Panel  3:  Family  risk 


Cohort  1 


Cohorts  1 and  2 


Average  Marginal  Average  Marginal 

Low  risk1  risk2  High  risk3  effect4  Low  Risk1  risk2  High  risk3  effect4 

Scale-Report (p-value)  (p-value)  (p-value) (p-value) (p-value)  (p-value)  (p-value)  (p-value) 

Student  Afraid  at  School-CR  (-)  -0.175*  -0.084  0.006  0.408*  -0.157*  -0.063  0.031  0.414* 


See  notes  at  end  of  table. 


Panel  4:  Community  risk 


Cohort  1 


Cohorts  1 and  2 


Scale-Report 

Low  risk1 
(p-value) 

Average 

risk2 

(p-value) 

High  risk3 
(p-value) 

Marginal 

effect4 

(p-value) 

Low  risk1 
(p-value) 

Average 

risk2 

(p-value) 

High  risk3 
(p-value) 

Marginal 

effect4 

(p-value) 

Problem  Behavior-PCR  (-) 

-0.045 

0.028 

0.101* 

0.109* 

-0.062 

0.028 

0.118* 

0.136’ 

(0.381) 

(0.446) 

(0.041) 

(0.032) 

(0.208) 

(0.442) 

(0.013) 

(0.005) 

Problem  Behavior-TRS  (-) 

0.087 

-0.001 

-0.089 

-0.132* 

0.087 

0.020 

-0.048 

-0.101 ' 

(0.268) 

(0.994) 

(0.243) 

(0.022) 

(0.254) 

(0.764) 

(0.511) 

(0.060) 

ADHD-Related  Behavior-TRS  (-) 

0.062 

-0.021 

-0.104 

-0.124* 

0.048 

-0.004 

-0.057 

-0.079 

(0.389) 

(0.722) 

(0.132) 

(0.029) 

(0.481) 

(0.938) 

(0.380) 

(0.130) 

Positive  School  Orientation-CR  (+) 

-0.086 

0.030 

0.146 

0.174** 

-0.051 

0.039 

0.129 

0.135’ 

(0.361) 

(0.719) 

(0.111) 

(0.007) 

(0.560) 

(0.607) 

(0.124) 

(0.023) 

Victimization  at  School-CR  (-) 

0.068 

-0.035 

-0.138* 

-0.154* 

0.054 

-0.026 

-0.106 

-0.120’ 

(0.298) 

(0.485) 

(0.030) 

(0.011) 

(0.432) 

(0.639) 

(0.106) 

(0.037) 

See  notes  at  end  of  table. 


Appendix  A:  Analysis  of  Cohorts  1 and  2 


A-36 


Table  A.16.  Cohort  1 versus  Cohorts  1 and  2:  Comparison  of  significant  impacts  by  subgroup — Continued 


Panel  5:  Initial  child  behavior  risk  as  reported  by  teacher 


Cohort  1 

Cohorts  1 and  2 

Low  risk1 

Average 

risk2 

High  risk3 

Marginal 

effect4 

Low  risk1 

Average 

risk2 

High  risk3 

Marginal 

effect4 

Scale-Report 

(p-value) 

(p-value) 

(p-value) 

(p-value) 

(p-value) 

(p-value) 

(p-value) 

(p-value) 

Normative  Beliefs  About  Aggression- 

0.127 

0.007 

-0.113 

-0.012** 

0.146A 

0.030 

-0.086 

-0.011** 

CR  (-) 

(0.134) 

(0.923) 

(0.191) 

(0.005) 

(0.068) 

(0.671) 

(0.285) 

(0.003) 

Problem  Behavior-CR  (-) 

0.075 

-0.007 

-0.089 

-0.008* 

0.077 

0.013 

-0.052 

-0.006A 

(0.348) 

(0.922) 

(0.275) 

(0.032) 

(0.320) 

(0.856) 

(0.507) 

(0.068) 

Problem  Behavior-TRS  (-) 

0.064 

-0.011 

-0.085 

-0.007* 

0.075 

0.022 

-0.030 

-0.005A 

(0.389) 

(0.872) 

(0.255) 

(0.029) 

(0.323) 

(0.746) 

(0.690) 

(0.099) 

Academic  Competence  and  Motivation- 

-0.108* 

-0.032 

0.044 

0.007** 

-0.119* 

-0.046 

0.027 

0.007** 

TRS  (+) 

(0.035) 

(0.455) 

(0.391) 

(0.008) 

(0.017) 

(0.277) 

(0.587) 

(0.006) 

Negative  School  Orientation-CR  (-) 

0.057 

-0.072 

-0.202* 

-0.013** 

0.066 

-0.047 

-0.160* 

-0.011** 

(0.467) 

(0.300) 

(0.013) 

(0.001) 

(0.364) 

(0.457) 

(0.031) 

(0.002) 

See  notes  at  end  of  table. 
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Table  A.16.  Cohort  1 versus  Cohorts  1 and  2:  Comparison  of  significant  impacts  by  subgroup — Continued 


Panel  6:  Initial  child  behavior  risk  as  reported  by  primary  caregiver 


Cohort  1 Cohorts  1 and  2 


Scale-Report 

Low  risk1 
(p-value) 

Average 

risk2 

(p-value) 

High  risk3 
(p-value) 

Marginal 

effect4 

(p-value) 

Low  risk1 
(p-value) 

Average 

risk2 

(p-value) 

High  risk3 
(p-value) 

Marginal 

effect4 

(p-value) 

Normative  Beliefs  About  Aggression- 

0.071 

-0.010 

-0.092 

-0.008* 

0.080 

0.010 

-0.061 

-0.007A 

CR  (-) 

(0.403) 

(0.893) 

(0.287) 

(0.036) 

(0.325) 

(0.989) 

(0.460) 

(0.062) 

Empathy-CR  (+) 

-0.083 

-0.007 

0.068 

0.008* 

-0.056 

-0.003 

0.049 

0.005 

(0.217) 

(0.897) 

(0.318) 

(0.042) 

(0.375) 

(0.948) 

(0.445) 

(0.138) 

Altruistic  Behavior-TRS  (+) 

-0.049 

0.029 

0.107 

0.008* 

-0.004 

0.060 

0.123 

0.006A 

(0.780) 

(0.864) 

(0.542) 

(0.050) 

(0.979) 

(0.693) 

(0.429) 

(0.095) 

Positive  Social  Behavior-PCR  (+) 

-0.086A 

-0.007 

0.072 

0.008* 

-0.080 

-0.015 

0.051 

0.006* 

(0.083) 

(0.852) 

(0.147) 

(0.015) 

(0.106) 

(0.701) 

(0.310) 

(0.037) 

Problem  Behavior-PCR  (-) 

-0.021 

0.036 

0.092A 

0.006 

-0.028 

0.037 

0.103* 

0.007* 

(0.691) 

(0.363) 

(0.080) 

(0.101) 

(0.571) 

(0.323) 

(0.043) 

(0.048) 

Engagement  with  Learning-CR  (+) 

-0.089 

-0.004 

0.081 

0.009* 

-0.071 

0.016 

0.102A 

0.009* 

(0.127) 

(0.925) 

(0.178) 

(0.028) 

(0.196) 

(0.707) 

(0.070) 

(0.018) 

Negative  School  Orientation-CR  (-) 

0.055 

0.055 

-0.178* 

-0.012** 

0.056 

-0.044 

-0.143A 

-0.010** 

(0.492) 

(0.492) 

(0.030) 

(0.002) 

(0.447) 

(0.501) 

(0.056) 

(0.006) 

See  notes  at  end  of  table. 
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Table  A.16.  Cohort  1 versus  Cohorts  1 and  2:  Comparison  of  significant  impacts  by  subgroup — Continued 


Panel  7:  Fidelity  of  implementation 


Cohort  1 

Cohorts  1 and  2 

Low  effect 
size 

Mixed  effect 
size 

High  effect 
size 

p-value  for 
test  of 

Low  effect 
size 

Mixed  effect 
size 

High  effect 
size 

p-value  for 
test  of 

Effect 

Effect 

Effect 

Effect 

Effect 

Effect 

Scale-Report 

size  p-value 

size  p-value 

size  p-value 

differences 

size  p-value 

size  p-value 

size  p-value 

differences 

Normative  Beliefs  About 

Aggression-CR  (-) 

0.06  0.419 

0.04  0.584  • 

-0.14  0.148 

0.068A 

0.11  0.174 

0.05  0.547 

-0.14  0.129 

0.042* 

* Significantly  different  from  zero  at  the  .05  level. 

**  Significantly  different  from  zero  at  the  .01  level. 

A Significantly  different  from  zero  at  the  .10  to  > .05  level. 

1 One  standard  deviation  below  the  mean  risk  level. 

2 At  the  mean  risk  level. 

3 One  standard  deviation  above  the  mean  risk  level. 

4 Change  in  impact  as  risk  level  increases  by  one  unit. 

NOTE:  Abbreviations  are 

CR:  Child  Report 

PCR:  Primary  Caregiver  Report 

TRS:  Teacher  Report  on  Student 

ADHD:  Attention  deficit  hyperactivity  disorder 

The  +/-  signs  in  parentheses  indicate  the  direction  of  a beneficial  outcome.  All  impact  estimates  were  calculated  using  regression  models  in  which  each  program  and  school  within  a 
program  was  weighted  equally.  The  standard  errors  of  all  estimates  account  for  design  effects  due  to  unequal  weighting  and  the  clustering  of  students  within  schools.  P-values  for  the 
effect  sizes  are  shown  within  the  parentheses  below  the  effect  size. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Growth  Analysis 

Regarding  the  growth  analysis,  none  of  the  18  estimated  combined  program  impacts  on  growth  in  fourth- 
grade  child  and  school  outcomes  was  statistically  significantly  different  from  zero  at  the  5 percent  level  (table 
A.  17).  The  lack  of  statistically  significant  beneficial  impact  estimates  at  the  combined -program  level  did  not 
appear  to  be  due  to  beneficial  impacts  in  some  programs  or  subgroups  that  were  offset  by  negative  impacts  in 
others. 
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Table  A.17.  Combined  Cohorts  1 and  2:  Impacts  on  growth  of  child  outcomes  from  combined-program  analysis 


Average  growth  in  the  score  per  year1 

Standard 


Scale-Report 

Mean  score  at 
implementation2 

Treatment 

group 

Control 

group 

Impact  on 
growth3 

Effect 

4 

size 

error  of 
impact 

p-value  of 
impact 

Social  and  Emotional  Competence  Domain 

Self-Efficacy  for  Peer  Interactions-CR  (+) 

2.92 

0.18 

0.18 

0.00 

0.00 

0.02 

0.854 

Normative  Beliefs  About  Aggression-CR  (-) 

1.24 

0.09 

0.08 

0.00 

0.01 

0.03 

0.885 

Empathy-CR  (+) 

2.41 

-0.16 

-0.15 

-0.01 

-0.02 

0.02 

0.705 

Behavior  Domain 

Altruistic  Behavior-CR  (+) 

1.45 

-0.25 

-0.24 

-0.02 

-0.02 

0.03 

0.549 

Altruistic  Behavior-PCR  (+) 

2.31 

-0.06 

-0.05 

-0.01 

-0.02 

0.02 

0.559 

Altruistic  Behavior-TRS  (+) 

1.40 

-0.02 

0.01 

-0.04 

-0.08 

0.05 

0.498 

Positive  Social  Behavior-PCR  (+) 

2.98 

0.05 

0.04 

0.01 

0.03 

0.01 

0.371 

Positive  Social  Behavior-TRS  (+) 

3.03 

-0.04 

0.01 

-0.05 

-0.07 

0.04 

0.205 

Problem  Behavior-CR  (-) 

0.25 

0.08 

0.08 

0.00 

0.01 

0.02 

0.827 

Problem  Behavior-PCR  (-) 

1.57 

-0.02 

-0.01 

-0.01 

-0.02 

0.01 

0.486 

Problem  Behavior-TRS  (-) 

1.38 

0.06 

0.07 

-0.01 

-0.02 

0.02 

0.654 

ADHD-Related  Behavior-TRS  (-) 

1.74 

-0.01 

0.00 

-0.01 

-0.02 

0.03 

0.624 

Academics  Domain 

Engagement  with  Learning-CR  (+) 

3.67 

-0.01 

-0.02 

0.01 

0.02 

0.02 

0.591 

Academic  Competence-TRS  (+) 

2.90 

0.04 

0.06 

-0.02 

-0.02 

0.03 

0.460 

See  notes  at  end  of  table. 
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Table  A.17.  Combined  Cohorts  1 and  2:  Impacts  on  growth  of  child  outcomes  from  combined-program  analysis — Continued 


Average  growth  in  the  score  per  year1 

Standard 


Scale-Report 

Mean  score  at 
implementation2 

T reatment 
group 

Control 

group 

Impact  on 
growth3 

Effect 

size4 

error  of 
impact 

p-value  of 
impact 

Perceptions  of  School  Climate  Domain 
Positive  School  Orientation-CR  (+) 

3.13 

-0.31 

-0.33 

0.02 

0.03 

0.03 

0.365 

Negative  School  Orientation-CR  (-) 

1.87 

0.08 

0.09 

-0.01 

-0.02 

0.02 

0.528 

Student  Afraid  at  School-CR  (-) 

2.42 

-0.13 

-0.08 

-0.05 

-0.05 

0.03 

0.102 

Victimization  at  School-CR  (-) 

0.79 

-0.08 

-0.03 

-0.04 

-0.05 

0.03 

0.134 

1 Pertains  to  the  estimated  slope  of  the  outcome  for  the  treatment  or  control  groups. 

2 The  average  score  at  implementation  was  calculated  across  treatment  and  control  groups,  using  regression  models  for  adjustment  on  covariates. 

3 Estimated  difference  between  the  slope  of  the  treatment  and  control  groups. 

4 The  slope  of  the  treatment  group  minus  the  slope  of  the  control  group  divided  by  the  standard  deviation  of  the  outcome  for  the  multisite  control  group  (the  standard  deviation  was 
calculated  without  accounting  for  school-level  clustering  or  regression  adjustments). 

NOTE:  Abbreviations  are 
CR:  Child  Report 
PCR:  Primary  Caregiver  Report 
TRS:  Teacher  Report  on  Student 
ADHD:  Attention  deficit  hyperactivity  disorder 

The  +/-  signs  in  parentheses  indicate  the  direction  of  a beneficial  outcome.  No  findings  were  found  statistically  significant  at  or  below  the  .05  level.  All  impact  estimates  were  calculated 
using  HLM  6.06.  Sample  weights  were  used  in  all  analyses  to  (1)  give  each  program  equal  weight  within  each  time  period,  (2)  give  each  school  equal  weight  in  each  program  (within 
each  time  period),  and  (3)  give  each  time  period  equal  weight  in  the  analysis. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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From  the  program-level  analysis,  12  impact  estimates  were  found  to  be  statistically  significant  out  of  126 
results  (more  than  the  6 that  would  be  expected  by  chance).  These  are  listed  in  table  A.  18,  which  shows  that  4 
were  beneficial  impacts  on  growth  in  student  outcomes,  and  8 were  detrimental  impacts. 


Table  A.18.  Combined  Cohorts  1 and  2:  Statistically  significant  impacts  from  the  growth  curve 
analyses  of  the  individual  programs 


Program 

Significant  beneficial  impacts 
(Report)  (Effect  size)  (p-value) 

Significant  detrimental  impacts 
(Report)  (Effect  size)  (p-value) 

Total 

4 

8 

ABC 

Positive  Social  Behavior 
(PCR)  (0.20)  (0.046) 

CSP 

Victimization  at  School 
(CR)  (-0.18)  (0.038) 

Academic  Competence 
(TRS)  (-0.15)  (0.025) 

Normative  Beliefs  About  Aggression 
(CR)  (0.33)  (0.031) 

LBW 

Problem  Behavior 
(CR)  (0.14)  (0.031) 

Normative  Beliefs  About  Aggression 
(CR)  (0.13)  (0.030) 

PA 

Altruistic  Behavior 
(PCR)  (0.19)  (0.021) 

PATHS 

4Rs 

Problem  Behavior 
(TRS)  (-0.21)  (0.048) 

SS 

Academic  Competence 
(TRS)  (-0.15)  (0.013) 

Positive  Social  Behavior 
(TRS)  (-0.35)  (0.008) 

Problem  Behavior 
(TRS)  (0.40)  (0.010) 

ADHD-Related  Behavior 
(TRS)  (0.28)  (0.000) 

NOTE:  Abbreviations  are 


ABC:  Academic  and  Behavioral  Competencies  Program 
CSP:  Competence  Support  Program 
LBW:  Love  In  a Big  World 
PA:  Positive  Action 

PATHS:  Promoting  Alternative  Thinking  Strategies 

4Rs:  The  4Rs  Program  (Reading,  Writing,  Respect,  and  Resolution) 

SS:  Second  Step 

CR:  Child  Report 

PCR:  Primary  Caregiver  Report 

TRS:  Teacher  Report  on  Student 

ADHD:  Attention  deficit  hyperactivity  disorder 

Blank  cell:  Finding  of  no  impact 

All  impact  estimates  were  calculated  using  HLM  6.06.  Sample  weights  were  used  in  all  analyses  to  (1 ) give  each  program  equal 
weight  within  each  time  period,  (2)  give  each  school  equal  weight  in  each  program  (within  each  time  period),  and  (3)  give  each  time 
period  equal  weight  in  the  analysis. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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The  subgroup  analyses  found  12  impact  estimates  that  significandy  differed  by  subgroup  out  of  the  126 
results  (more  than  the  6 that  would  be  expected  by  chance).  Table  A.  19  shows  the  statistically  significant 
impacts  for  the  subgroups.  The  gender  analysis  found  1 significant  beneficial  impact  for  girls:  a decline  in 
Negative  School  Orientation.  New  entrants  had  detrimental  impacts  for  growth  in  both  Positive  School 
Orientation  and  Negative  School  Orientation.  For  the  initial  risk  analyses,  9 impacts  were  found  to  be 
significant,  with  8 showing  beneficial  impacts  on  growth  for  high-risk  students  and  1 showing  detrimental 
impacts  on  high-risk  students.  Subgroup-specific  regression  results  showed  that  2 of  the  beneficial  impacts 
were  actually  due  to  detrimental  growth  effects  for  low-risk  students  (Academic  Competence  in  regards  to 
child  behavior  risk  as  reported  by  both  the  teacher  and  the  primary  caregiver).  In  addition,  they  showed  that 
the  one  finding  of  a detrimental  impact  on  high-risk  students  was  due  to  a beneficial  impact  on  low-risk 
students  (Problem  Behavior  reported  in  the  Primary  Caregiver  Report  in  regards  to  community  risk). 
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Table  A.19.  Combined  Cohorts  1 and  2:  Statistically  significant  impacts  from  the  growth  curve 
analyses  of  the  subgroups 


Subgroup 

Outcome 

Impact  on  subgroup 

Gender 

Negative  School  Orientation 
(p-value  of  impact  = 0.033) 

Beneficial  to  girls 

New  entrants/stayers 

Positive  School  Orientation 
(p-value  of  impact  = 0.049) 

Detrimental  to  new  entrants 

Negative  School  Orientation 
(p-value  of  impact  = 0.034) 

Socioeconomic  risk 

Academic  Competence 
(Interaction  term  = 0.05) 
(p-value  of  impact  = 0.032) 

Beneficial  to  high  risk 

Family  risk 

Victimization  at  School 
(Interaction  term  = -0.20) 
(p-value  of  impact  = 0.026) 

Beneficial  to  high  risk 

Community  risk 

Problem  Behavior  (PCR) 
(Interaction  term  = 0.04) 
(p-value  of  impact  = 0.013) 

Detrimental  to  high  risk  due  to 
beneficial  for  low  risk 

Problem  Behavior  (TRS) 
(Interaction  term  = -0.05) 
(p-value  of  impact  = 0.025) 

Beneficial  to  high  risk 

Positive  School  Orientation 
(Interaction  term  = 0.1 1) 
(p-value  of  impact  = 0.001) 

Beneficial  to  high  risk 

Child  behavior  risk  (TRS) 

Academic  Competence 
(Interaction  term  = 0.03) 
(p-value  of  impact  = 0.002) 

Beneficial  to  high  risk  due  to 
detrimental  for  low  risk 

Positive  Social  Behavior  (PCR) 
(Interaction  term  = 0.02) 
(p-value  of  impact  = 0.016) 

Beneficial  to  high  risk 

Child  behavior  risk  (PCR) 

Academic  Competence 
(Interaction  term  = 0.02) 
(p-value  of  impact  = 0.010) 

Beneficial  to  high  risk  due  to 
detrimental  for  low  risk 

Positive  Social  Behavior  (PCR) 
(Interaction  term  = 0.02) 
(p-value  of  impact  = 0.002) 

Beneficial  to  high  risk 

NOTE:  Abbreviations  are 


PCR:  Primary  Caregiver  Report 
TRS:  Teacher  Report  on  Student 

All  impact  estimates  were  calculated  using  HLM  6.06.  Sample  weights  were  used  in  all  analyses  to  (1 ) give  each  program  equal 
weight  within  each  time  period,  (2)  give  each  school  equal  weight  in  each  program  (within  each  time  period),  and  (3)  give  each  time 
period  equal  weight  in  the  analysis. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Summary 

As  a result  of  an  effort  to  increase  the  sample  size  and  corresponding  statistical  power  of  the  SACD 
evaluation,  a second  cohort  of  12  schools  was  added  in  Year  2.  Six  of  the  schools  implemented  one  of  four 
SACD  programs  while  the  other  six  acted  as  their  control  schools.  Overall,  the  analyses  of  the  combined 
cohort  sample  produced  results  similar  to  those  of  the  analyses  of  the  Cohort  1 data. 

Responses  from  students,  primary  caregivers,  and  teachers  from  the  original  83  and  the  12  new  schools 
during  the  year  students  were  in  the  fourth  grade  were  combined.  The  combined  treatment  and  control 
groups  remained  similar,  as  they  were  in  the  Cohort  1 study,  although  the  Cohort  2 students  were  significantly 
different  from  the  Cohort  1 students  in  that  they  came  from  more  disadvantaged  households  and 
communities.  The  analyses  done  with  the  fourth-grade  Cohort  1 students  were  replicated  with  the  students  of 
Cohorts  1 and  2 combined. 

The  results  of  the  analysis  of  teachers’  activities  and  professional  development,  pooled  across  programs,  were 
similar  to  the  results  for  the  Cohort  1 data.  These  results  showed  that  statistically  significant  larger 
percentages  of  treatment  group  teachers  than  control  group  teachers  reported  engaging  in  activities  to 
promote  social  and  character  development,  using  materials  and  strategies  to  promote  social  and  character 
development,  and  participating  in  related  professional  development.  Treatment  group  teachers  were  no  more 
likely  to  report  changes  in  the  school  environment  or  in  teacher  and  staff  s attitudes  that  might  be  conducive 
to  promoting  social  and  character  development. 

The  findings  from  the  combined  cohort  study  were  similar  to  those  from  the  Cohort  1 study.  In  the  year-by- 
year analysis,  the  seven  SACD  combined  programs  had  no  significant  impact  on  the  20  outcomes  as 
compared  to  the  impact  on  one  outcome  (Student  Support  for  Teachers)  found  in  the  analysis  of  Cohort  1. 
Regarding  the  domains,  the  analysis  found  a significant  detrimental  effect  on  the  domain  of  Social  and 
Emotional  Competence.  The  lack  of  impacts  at  the  combined-program  level  was  not  due  to  beneficial 
impacts  of  some  programs  being  offset  by  detrimental  impacts  of  other  programs.  For  18  of  the  20 
outcomes,  there  were  no  statistically  significant  differences  in  impacts  across  the  programs  (the  two 
exceptions  were  due  to  significant  positive  impacts  by  the  ABC  Program). 

The  individual  program  impacts  for  the  four  programs  that  had  Cohort  2 students  showed  little  or  no 
improvement  over  those  of  the  Cohort  1 analysis.  One  program  lost  a significant  beneficial  impact  and  gained 
a significant  detrimental  impact  (CSP).  One  program  lost  a substantively  important  beneficial  impact  and  a 
substantively  important  detrimental  impact  and  gained  a substantively  important  detrimental  impact  (LBW). 
One  program  gained  a substantively  important  beneficial  impact  (PATHS),  and  one  program  lost  a significant 
detrimental  effect  (ABC). 

The  findings  from  subgroup  analyses  of  the  combined  cohorts  were  the  same  for  14  out  of  22  impacts  that 
were  found  significant  in  the  Cohort  1 subgroup  analysis.  The  differences  between  the  findings  were  primarily 
that  the  analyses  of  the  combined  cohorts  provided  less  evidence  of  more  beneficial  impacts  for  higher  risk 
students  and  mixed  evidence  regarding  impacts  for  stayers  versus  new  entrants. 

For  the  growth  curve  analysis,  the  results,  though  not  directly  comparable  to  the  Cohort  1 analysis,  were 
similar.  The  seven  SACD  programs,  combined,  had  no  significant  impact  on  the  1 8 outcomes.  The  individual 
program  analysis  found  a small  number  of  significant  impacts  on  student  growth  in  outcomes  but  two-thirds 
of  these  were  detrimental  impacts.  One  difference  was  that  the  initial  risk  analyses  had  slightly  more  findings 
than  would  be  expected  by  chance,  and  the  majority  of  these  showed  more  beneficial  impacts  for  higher  risk 
students. 
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Table  A.20.  Sample  size  ranges  for  outcome  analyses 

Minimum 

Maximum 

Outcome  analysis  Group 

Report 

of  range 

of  range 

Combined  programs  Treatment 

CR 

2,389 

2,409 

PCR 

1,833 

1,859 

TRS 

2,450 

2,498 

TRCS 

478 

479 

Control 

CR 

2,098 

2,121 

PCR 

1,651 

1,671 

TRS 

2,208 

2,221 

TRCS 

458 

459 

Individual  programs 

ABC  Treatment 

CR 

258 

259 

PCR 

182 

185 

TRS 

259 

271 

TRCS 

55 

55 

Control 

CR 

342 

345 

PCR 

257 

257 

TRS 

357 

357 

TRCS 

72 

72 

CSP  Treatment 

CR 

428 

436 

PCR 

323 

326 

TRS 

420 

444 

TRCS 

81 

81 

Control 

CR 

406 

416 

PCR 

339 

341 

TRS 

433 

433 

TRCS 

82 

82 

LBW  Treatment 

CR 

359 

360 

PCR 

312 

316 

TRS 

379 

387 

TRCS 

75 

76 

Control 

CR 

282 

284 

PCR 

225 

227 

TRS 

295 

297 

TRCS 

56 

56 

PA  Treatment 

CR 

280 

284 

PCR 

212 

214 

TRS 

289 

302 

TRCS 

43 

43 

Control 

CR 

220 

225 

PCR 

181 

183 

TRS 

249 

251 

TRCS 

39 

39 

See  notes  at  end  of  table. 
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Table  A.20.  Sample  size  ranges  for  outcome  analyses- 

-Continued 

Outcome  analysis  Group 

Report 

Minimum 
of  range 

Maximum 
of  range 

PATHS  Treatment 

CR 

305 

308 

PCR 

233 

239 

TRS 

315 

315 

TRCS 

74 

74 

Control 

CR 

262 

263 

PCR 

198 

199 

TRS 

271 

271 

TRCS 

76 

76 

4Rs  Treatment 

CR 

372 

377 

PCR 

241 

246 

TRS 

399 

404 

TRCS 

90 

91 

Control 

CR 

292 

299 

PCR 

206 

209 

TRS 

307 

320 

TRCS 

82 

82 

SS  Treatment 

CR 

383 

387 

PCR 

330 

335 

TRS 

381 

383 

TRCS 

59 

60 

Control 

CR 

288 

291 

PCR 

245 

255 

TRS 

292 

293 

TRCS 

51 

52 

Subgroups 

Gender  Boys 

CR 

2,124 

2,144 

PCR 

1,643 

1,669 

TRS 

2,240 

2,261 

Girls 

CR 

2,363 

2,386 

PCR 

1,825 

1,845 

TRS 

2,400 

2,441 

New  entrants  Stayers 

CR 

3,461 

3,495 

PCR 

2,699 

2,731 

TRS 

3,607 

3,651 

New  entrants 

CR 

1,026 

1,036 

PCR 

785 

799 

TRS 

1,051 

1,069 

Initial  socioeconomic  risk 

CR 

4,062 

4,106 

PCR 

3,407 

3,448 

TRS 

4,218 

4,261 

See  notes  at  end  of  table. 
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Table  A.20.  Sample  size  ranges  for  outcome  analyses— 

-Continued 

Outcome  analysis  Group 

Report 

Minimum 
of  range 

Maximum 
of  range 

Initial  family  risk 

CR 

3,068 

3,099 

PCR 

2,531 

2,561 

TRS 

3,195 

3,222 

Initial  community  risk 

CR 

3,019 

3,051 

PCR 

2,501 

2,529 

TRS 

3,140 

3,166 

Initial  child  behavior  risk 

(Teacher  Report) 

CR 

3,288 

3,322 

PCR 

2,582 

2,610 

TRS 

3,430 

3,468 

Initial  child  behavior  risk 
(Primary  Caregiver 

Report) 

CR 

3,050 

3,082 

PCR 

2,522 

2,551 

TRS 

3,179 

3,206 

Fidelity  of  implementation  High 

CR 

1,749 

1,767 

PCR 

1,423 

1,442 

TRS 

1,788 

1,813 

Mixed 

CR 

848 

851 

PCR 

677 

684 

TRS 

893 

904 

Low 

CR 

1,887 

1,909 

PCR 

1,381 

1,402 

TRS 

1,974 

2,001 

NOTE:  Abbreviations  are 

ABC:  Academic  and  Behavioral  Competencies  Program 
CSP:  Competence  Support  Program 
LBW:  Love  In  a Big  World 
PA:  Positive  Action 

PATHS:  Promoting  Alternative  Thinking  Strategies 

4Rs:  The  4Rs  Program  (Reading,  Writing,  Respect,  and  Resolution) 

SS:  Second  Step 
CR:  Child  Report 
PCR:  Primary  Caregiver  Report 
TRS:  Teacher  Report  on  Student 
TRCS:  Teacher  Report  on  Classroom  and  School 
SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Technical  Notes  on  the  Development  of  Outcome  Measures, 
Selection  of  Covariates,  Construction  of  Sample  Weights, 

and  Sensitivity  Analyses 


This  technical  appendix  supplements  the  Study  Design  and  Methodology  section  of  chapter  1 with  additional 
details  on  the  methods  used  for  the  SACD  impact  analysis.  It  provides  information  about  the  development  of 
the  outcome  measures,  the  selection  of  covariates  that  were  included  in  the  regression  models,  the  sample 
weights,  and  the  sensitivity  analysis  used  to  examine  the  robustness  of  the  impact  estimates  to  alternative 
parameter  assumptions  and  specifications. 

Development  of  the  Outcome  Measures 

This  section  describes  the  procedures  used  to  create  and  validate  the  survey-based  child  and  school  outcome 
measures.  The  majority  of  the  discussion  concerns  the  18  student-level  outcome  measures,  with  the  2 school- 
level  outcome  measures  discussed  at  the  end  of  the  section.  The  development  of  the  outcome  measures  was 
directed  by  the  Social  and  Character  Development  (SACD)  Statistical  Workgroup,  which  was  made  up  of 
representatives  from  the  National  Center  for  Education  Research,  Institute  of  Education  Sciences  (IES),  U.S. 
Department  of  Education;  the  Division  of  Violence  Prevention  in  the  National  Center  for  Injury  Prevention 
and  Control,  Centers  for  Disease  Control  and  Prevention  (CDC);  SACD  grantees;  Mathematica  Policy 
Research,  Inc.  (MPR);  and  the  University  of  Missouri-St.  Louis.  Each  outcome  measure  is  a scale  composite 
of  multiple  Likert-type  items,  derived  from  initial  (fall  2004)  survey  responses  to  the  Child  Report,  Primary 
Caregiver  Report,  and  the  Teacher  Report  on  Student.1  For  example,  the  five  student-level  measures  created 
from  the  Teacher  Report  on  Student  are  Positive  Social  Behavior,  Problem  Behavior,  Altruistic  Behavior, 
ADHD-Related  Behavior,  and  Academic  Competence  and  Motivation.  A list  of  all  scales  for  each  respondent 
group  is  provided  in  table  B.l. 

The  surveys  were  either  self-administered  or,  for  some  primary  caregivers,  administered  as  a computer- 
assisted  telephone  interview.  The  items  included  in  the  initial  versions  of  the  teacher-,  primary  caregiver-,  and 
child-report  surveys  were  largely  taken  from  instruments  used  in  previous  research.  Four  measures  (child 
responsibility,  child  and  teacher  perceptions  of  school  safety,  and  primary  caregiver  perceptions  of 
community  resources)  were  generated  by  SACD  Consortium  members  due  to  a lack  of  adequate  existing 
measures  of  those  constructs.  Preliminary  analyses  of  the  measures  administered  in  the  initial  wave  of  data 
collection  revealed  that,  although  most  of  the  scales  performed  adequately  and  as  expected,  some  predefined 
scales  had  poor  internal  consistency  (Cronbach’s  alphas  < 0.60  for  primary  caregiver  report  on  the  Behavior 
Assessment  System  for  Children  [BASC]  Conduct  Problems  Subscale  and  child  report  on  the  Nonconflict 
Subscale  items  of  the  Children’s  Self-Efficacy  for  Peer  Interaction  Scale).  Furthermore,  a small  number  of  the 
scale  or  subscale  scores  were  highly  intercorrelated  (for  example,  r>  0.80  for  the  correlation  between  teacher 
report  on  the  Responsibility  Scale  and  teacher  report  on  the  Social  Competence  Scale).  Consequently,  the 
SACD  Statistical  Workgroup  conducted  more  rigorous  analyses  of  the  items  to  determine  whether  a set  of 
outcome  measures  with  better  psychometric  characteristics  could  be  derived. 


1 In  fall  2004,  data  collection  began  at  an  original  cohort  of  schools  and  focused  on  students  who  were  beginning  third 
grade.  Students  at  these  schools  were  followed  through  spring  of  fifth  grade.  A second  cohort  of  schools  was 
subsequently  included  in  the  study;  data  collection  at  these  schools  focused  on  students  who  started  third  grade  in  fall 
2005;  these  students  were  followed  through  the  spring  of  fourth  grade.  The  analyses  discussed  here  pertain  to  the  first 
cohort  of  schools  and  students. 
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Table  B.1 . Scales  and  internal  consistency  for  child  outcomes,  by  report  type 


Scale  (Cronbach’s  a) 

Teacher  Report  on  Student 

Child  Report 

Positive  Social  Behavior  (0.97) 

Positive  School  Orientation  (0.86) 

Problem  Behavior  (0.95) 

Engagement  with  Learning  (0.84) 

Altruistic  Behavior  (0.89) 

Student  Afraid  at  School  (0.79) 

ADHD-Related  Behavior  (0.91) 

Negative  School  Orientation  (0.78) 

Academic  Competence  and  Motivation  (0.95) 

Self-Efficacy  for  Peer  Interaction  (0.83) 
Normative  Beliefs  About  Aggression  (0.83) 

Primary  Caregiver  Report 

Empathy  (0.78) 

Positive  Social  Behavior  (0.93) 

Altruistic  Behavior  (0.88) 

Problem  Behavior  (0.86) 

Problem  Behavior  (0.86) 

Altruistic  Behavior  (0.88) 

Victimization  at  School  (0.86) 

NOTE:  Internal  consistency  is  Cronbach’s  cr  (alpha),  measured  using  a random  sample  of  one-half  of  the  respondents  to  the  fall 
2004  survey  administration.  Consistency  remained  similarly  high  at  all  waves.  Listwise  deletion  of  cases  with  missing  data  was  used 
in  all  analyses.  ADHD  is  the  abbreviation  for  attention  deficit  hyperactivity  disorder. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 


The  approach  the  SACD  Statistical  Workgroup  used  was,  first,  to  conduct  exploratory  factor  analyses.  Using 
half  of  the  sample,  selected  randomly,  researchers  looked  at  the  individual  items  of  the  fall  2004  survey 
separately  for  each  group  of  respondents  (teachers,  primary  caregivers,  and  children).  These  analyses  were 
conducted  separately  for  each  group  of  respondents  because  the  survey  instruments  for  teachers,  primary 
caregivers,  and  children  contained  different  sets  of  items.  Although  a small  number  of  measures  were 
administered  to  more  than  one  respondent  group  (see  below),  most  of  the  measures  were  administered  to 
only  a single  group  of  respondents.  Potential  outcome  scales  identified  in  these  exploratory  analyses  were 
then  subjected  to  confirmatory  factor  analyses  with  the  remaining  half  of  the  sample  (hereafter  referred  to  as 
the  “validation  sample”)  using  structural  equation  modeling  procedures. 

For  each  of  the  three  survey  respondent  groups,  a set  of  exploratory  factor  analyses  was  undertaken  using  the 
principal  axis  method.  Based  on  both  conceptual  and  empirical  (such  as  eigenvalues)  criteria,  varying  numbers 
of  factors  were  extracted  and  rotated  in  these  analyses,  and  then  compared  with  respect  to  the  conceptual 
clarity  of  the  factors,  the  nature  and  extent  of  cross-loading  of  items,  the  extent  to  which  factors  were  defined 
by  a very  small  number  of  items  (or  single  items),  and  parsimony  (the  absence  of  multiple  factors  appearing 
to  assess  the  same  basic  construct).  On  the  basis  of  these  analyses,  it  was  decided  that  the  item  responses  of 
teachers,  primary  caregivers,  and  children  were  optimally  represented  by  5,  3,  and  10  underlying  factors, 
respectively.  The  names  and  internal  consistencies  of  the  scales  are  listed  in  table  B.l.2 

Confirmatory  Factor  Analyses  of  Student-Level  Outcome  Measures 

The  factors  identified  in  the  exploratory  analyses  were  subjected  to  empirical  confirmation  using  the 
validation  samples.  Analyses  of  model  fit  were  conducted  using  LISREL  software  for  the  analysis  of 
covariance  structures.  The  focus  was  on  two  measures  of  model  fit:  the  comparative  fit  index  (CF1)  and  the 
root  mean  square  error  of  approximation  (RMSEA).  The  CF1  runs  from  0 to  1 and  gauges  the  percentage 
improvement  in  reproducing  the  input  covariance  matrix  obtained  by  using  the  proposed  measurement 
model  as  compared  to  a null  model  that  assumes  no  shared  variance  (and  thus  no  latent  factors).  Higher 


2 A small  number  of  items  were  dropped  from  further  analyses  because  they  did  not  load  on  any  of  the  identified  factors 
(loadings  were  not  > 0.30  on  any  factor).  Specifically,  8 of  91  items  in  the  child  survey  (9%),  3 of  74  items  in  the  teacher 
survey  (4%),  and  6 of  59  items  in  the  primary  caregiver  survey  (10%)  were  dropped  from  analysis.  Items  that  cross- 
loaded  on  other  factors  (an  item  was  considered  to  cross-load  on  two  or  more  factors  when  factor  loadings  were  greater 
than  0.30)  were  included  on  the  factor  for  which  the  loading  was  stronger. 
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values  represent  improved  fit  and,  while  all  fit  standards  are  somewhat  arbitrary,  researchers  have  variously 
suggested  either  .90  or  .95  as  cutoffs  for  good  fit.  The  RMSEA  is  an  index  of  lack  of  fit  (based  on  the  %2)  per 
degree  of  freedom.  The  RMSEA  also  runs  from  0 to  1,  but  higher  values  reflect  poorer  fit.  Suggested  cutoffs 
for  acceptable  fit  vary  from  .08  to  .10. 

For  the  teacher,  the  primary  caregiver,  and  the  child  respondents,  the  hypothesized  factor  representations 
provided  a good  fit  to  the  data,  as  estimated  using  LISREL  structural  equations  modeling  software.  As  shown 
in  table  B.2,  for  the  71  items  in  the  teacher  survey,  the  confirmatory  factor  analysis  of  five  latent  factors 
yielded  a CFI  value  of  0.98  and  an  RMSEA  of  0.090  (90%  confidence  interval  [Cl]  = 0.089,  0.091).  For  the  53 
primary  caregiver  items,  the  confirmatory  factor  analysis  of  three  latent  factors  yielded  a CFI  of  0.94  and  an 
RMSEA  of  0.087  (90%  Cl  = 0.086,  0.088);  for  the  83  child  items,  the  confirmatory  factor  analysis  of  10  latent 
factors  yielded  a CFI  of  0.91  and  an  RMSEA  of  0.060  (90%  Cl  = 0.059,  0.061). 
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Table  B.2.  Fit  indexes  from  confirmatory  factor  analyses,  five-factor  measurement  model:  First 
four  waves  of  data  collection,  by  respondent  type 


Respondent  type/Time  of  data  collection 

Sample  size 

RMSEA 

CFI 

Teachers  (5  factors) 

Initial 

2,040 

0.090 

0.980 

Follow-up  1 

3,538 

0.114 

0.967 

Follow-up  2 

3,410 

0.102 

0.969 

Follow-up  3 

3,580 

0.107 

0.970 

Follow-up  4 

3,178 

0.110 

0.969 

Primary  Caregivers  (3  factors) 

Initial 

1,389 

0.087 

0.940 

Follow-up  1 

2,643 

0.079 

0.956 

Follow-up  2 

2,745 

0.081 

0.957 

Follow-up  3 

2,639 

0.082 

0.954 

Follow-up  4 

2,355 

0.084 

0.957 

Children  (10  factors) 

Initial 

1,314 

0.060 

0.910 

Follow-up  1 

3,074 

0.064 

0.943 

Follow-up  2 

3,247 

0.067 

0.947 

Follow-up  3 

3,307 

0.066 

0.952 

Follow-up  4 

3,412 

0.066 

0.954 

NOTE:  Abbreviations  are 
CFI:  Comparative  fit  index 
RMSEA:  Root  mean  square  error  of  approximation 

Listwise  deletion  of  cases  with  missing  data  was  used  in  all  analyses.  Confirmatory  factor  models  were  estimated  using  the  LISREL 
structural  equation  modeling  package,  with  latent  factor  variances  fixed  to  1 .0  to  scale  the  factors.  Initial  data  were  collected  in  fall 
2004.  Follow-up  1 was  in  spring  2005,  Follow-up  2 was  in  fall  2005,  Follow-up  3 was  in  spring  2006,  and  Follow-up  4 was  in  spring 
2007.  The  fall  2004  results  are  from  analyses  of  random  samples  of  one-half  of  the  fall  2004  respondents  of  each  respective 
respondent  type.  Results  for  the  follow-ups  are  derived  from  analyses  of  full  samples  (after  listwise  deletion). 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 


Robustness  Checks 

The  analyses  of  the  fall  2004  data  indicated  that,  for  the  sample  as  a whole,  the  child  outcome  variables  could 
be  adequately  represented  by  a relatively  small  number  of  scales.  Further  analyses  were  performed  to  test  the 
robustness  of  the  model  (1)  for  major  demographic  subgroups,  (2)  across  samples  at  different  study  sites,  (3) 
to  treatment  of  missing  data,  (4)  to  different  estimation  approaches,  and  (5)  across  different  survey  waves. 

Demographic  and  Site  Checks 

A series  of  multigroup  confirmatory  analyses  were  conducted  to  investigate  whether  the  identified 
measurement  models  held  across  major  child  demographic  groups  (gender  and  race/ethnicity)  and  across  the 
seven  individual  SACD  programs.  In  each  case,  a measurement  model  in  which  all  factor  loadings  and  factor 
covariances  were  constrained  to  be  equal  across  groups  (but  error  variances  were  allowed  to  vary)  provided  a 
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good  fit  to  the  multigroup  data.3  The  basic  measurement  model  thus  proved  to  be  valid  across  child  gender, 
across  child  race/ ethnicity,  and  for  each  of  the  seven  SACD  programs. 

Missing  Items 

A second  set  of  confirmatory  analyses  were  conducted  to  examine  whether  the  measurement  models  were 
robust  to  differences  in  the  treatment  of  missing  item-response  data.  Primary  analyses  were  restricted  to 
observations  with  no  missing  items.  Although  preliminary  analyses  indicated  that  missing  responses  to 
individual  survey  items  were  infrequent  among  each  of  the  respondent  groups  (no  more  than  5%  of  the 
responses  were  missing  for  any  survey  item),  listwise  deletion  had  a cumulative  effect  of  excluding  one-third 
or  more  of  the  respondents  in  each  group.  Analyses  of  missing  data  showed  that  respondents  with  valid 
scores  on  the  outcome  measures  were  similar  to  those  with  missing  data,  suggesting  that  data  were  missing  at 
random.  Nonetheless,  to  investigate  potential  biases  due  to  exclusion  of  cases  with  missing  responses  to 
survey  items,  the  confirmatory  factor  analyses  were  re-run  with  missing  item  responses  imputed  using  the 
Expectation-Maximization  (EM)  algorithm.4  For  each  respondent  group,  the  use  of  EM  imputation  resulted 
in  a fit  as  good  as  or  better  than  that  of  models  on  samples  that  used  listwise  deletion. 

Estimation  Procedures 

Approaches  to  evaluating  model  fit  vary  across  software  packages.  LISREL  uses  a “weighted  least  squares” 
chi-square  independence  model.  By  contrast,  another  structural  equation  modeling  package,  EQS,  uses  a 
“minimum  fit  function”  chi-square  independence  model.  For  equivalent  models,  LISREL  and  EQS  should 
produce  the  same  normal  theory  chi-square  and  RMSEA  values  but  would  produce  different  measures  of 
relative  fit,  such  as  CFI,  due  to  the  use  of  different  chi-square  independence  models.  Analyses  performed 
using  EQS  showed  that,  as  expected,  LISREL  yielded  a higher  CFI  for  each  model  than  EQS.5  However,  the 
other  fit  indexes,  which  are  not  based  on  choice  of  chi-square  independence  model,  were  actually  better  as 
estimated  by  EQS  than  LISREL,  and  both  software  packages  indicated  that  the  measurement  models 
provided  a good  fit  to  the  data. 

Successive  Data  Waves 

Finally,  it  is  possible  that  the  SACD  programs  might  influence  not  only  the  mean  scores  on  the  measured 
outcomes  but  also  the  correlations  among  the  measured  outcomes,  and  thus  have  an  impact  on  the  quality  of 
the  measurement  model.  Analyses  were  performed  to  test  the  continuing  fit  of  the  model  with  each  new  wave 
of  data.  In  addition  to  presenting  measures  of  model  fit  from  the  initial  survey  wave,  table  B.2  also  presents 
corresponding  measures  for  each  of  the  succeeding  follow-up  waves  for  the  original  cohort.  Those  results 
confirm  that  the  model  has  held  up  well  over  time.  There  was  minimal  variation  in  either  of  the  fit  statistics 
across  survey  waves. 


3 All  multigroup  confirmatory  factor  analyses  were  conducted  using  the  random  half-samples  of  respondents  used  in  the 
initial  confirmatory  analyses. 

4 The  EM  algorithm  (see  Dempster,  Laird,  and  Rubin  1977)  is  an  iterative,  maximum -likelihood  procedure  that  uses 
information  from  observed  data  to  successively  estimate,  evaluate,  and  re-estimate  values  for  missing  data  until  it 
converges  on  a unique  maximum-likelihood  estimate. 

5 The  independence  model  specifies  that  the  covariance  matrix  of  the  observed  variables  is  a diagonal  matrix  with  all 
observed  variables  uncorrelated.  Although  the  computational  formulas  are  complicated  (see  Joreskog  et  al.  2003),  the 
weighted  least  squares  independence  model  chi-square  (LISREL  default)  yields  larger  values  than  the  minimum  fit 
function  independence  model  chi-square  (EQS  default).  As  a result,  incremental  fit  indexes  such  as  the  CFI  will  be  larger 
in  LISREL  than  in  EQS,  because  all  such  indexes  are  computed  as  some  function  of  the  chi  square  (1  — [model  chi- 
square/independence  chi-square]).  As  should  be  the  case,  when  the  same  chi-square  independence  model  is  used  to 
evaluate  model  fit  in  LISREL  and  EQS,  the  two  software  packages  produce  virtually  identical  estimates  of  CFI  and  all 
other  model  fit  indexes  that  are  evaluated  against  the  independence  model. 
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Construct  and  Reporter-Based  Variance 

The  measurement  models  described  above  all  derived  measures  separately  for  each  respondent  group 
(children,  primary  caregivers,  and  teachers).  However,  comparable  measures  were  found  among  the 
respondent  groups — such  as  Altruistic  Behavior,  Positive  Social  Behavior,  and  Problem  Behavior.  A series  of 
exploratory  “multitrait,  multirespondent”  (MTMR)  analyses  were  undertaken  to  investigate  the  extent  of 
convergence  across  respondents  in  these  commonly  measured  outcomes — whether  responses  predominantly 
reflected  common  perceptions  of  child  behaviors  across  respondents  or  whether  they  were  largely 
idiosyncratic  to  the  respondent.  That  is,  these  efforts  represented  an  attempt  to  distinguish  construct  variance 
from  systematic  variance  due  to  the  respondent  (as  well  as  random  measurement  error). 

Systematic  differences  between  respondents  may  represent  reporter  bias  or  may  reflect  differences  associated 
with  the  different  types  of  information  available  to  each  respondent  when  making  judgments.  For  example, 
primary  caregivers  and  teachers  differ  in  their  overall  familiarity  with  a child  and  in  the  nature  of  the 
situations  in  which  they  are  able  to  observe  the  child’s  behavior.  These  differences  in  available  information 
are  quite  likely  to  influence  their  judgments  about  the  child’s  characteristics.  Thus  it  is  important  to 
acknowledge  that  systematic  variance  between  respondents  might  not  necessarily  indicate  error  in 
observation,  recollection,  or  reporting,  but  rather  they  could  reflect  real  differences  in  child  behavior  in 
different  environments  and  contexts. 

The  SACD  Statistical  Workgroup  conducted  multiple  analyses  with  “construct”  latent  variables  (e.g.,  Problem 
Behavior,  Altruistic  Behavior)  and  “reporter”  latent  variables  (e.g.,  primary  caregiver).  The  central  finding  of 
these  MTMR  analyses  was  that  for  the  set  of  outcomes  measured,  there  was  substantial  variance  attributable 
to  both  the  respondent  and  the  constructs  being  measured.  Thus,  for  example,  the  correlations  between  a 
single  respondent’s  assessments  across  constructs — such  as  a teacher’s  perceptions  of  a child’s  Problem 
Behavior  and  Altruistic  Behavior — were  similar  to  the  correlations  between  respondents  for  the  same 
construct — such  as  the  teacher’s  assessment  of  that  child’s  Problem  Behavior  and  the  corresponding 
assessment  by  the  primary  caregiver.  In  addition,  the  path  coefficients  for  the  construct  and  reporter  latent 
variables  were  of  generally  similar  magnitude  to  one  another.  The  data  used  in  these  analyses  do  not  permit 
determination  of  whether  the  latent  reporter  effects  reflected  bias  on  the  part  of  respondents  or  differences  in 
the  information  available  to  different  respondents. 

Psychometric  Properties  of  School-Level  Outcome  Measures 

In  addition  to  the  individual,  student -level  outcome  measures  described  above,  two  school-level  measures  of 
possible  SACD  program  outcomes  were  included  in  the  impact  analyses.  These  measures  were  included  in 
separate  teacher  surveys  (Teacher  Report  on  Classroom  and  School)  administered  in  fall  2004  and  each 
follow-up  period,  and  the  measures  focused  on  assessments  of  the  school  environment.  One  of  the  measures, 
Student  Support  for  Teachers,  assessed  teacher  perceptions  of  student  positive  and  negative  behavior  and 
teacher-student  relations;  the  other,  Feelings  of  Safety,  assessed  teacher  perceptions  of  the  extent  to  which 
students  were  not  fearful  of  being  psychologically  or  physically  harmed  at  school.  Principal  components 
analyses  revealed  that  both  measures  were  unidimensional,  with  all  items  having  large  positive  loadings  (> 
.69)  on  the  first  unrotated  component  and  the  component  accounting  for  62  percent  to  77  percent  of  the 
variance  in  item  responses.  Similarly,  internal  consistency  analyses  revealed  that  both  measures  were  highly 
reliable,  with  alpha  coefficients  between  .89  and  .91  at  each  survey  wave  for  the  Student  Support  for  Teachers 
index  and  between  .88  and  .90  at  each  wave  for  the  Feelings  of  Safety  index. 

Selection  of  Covariates 

This  section  provides  details  on  the  selection  of  covariates  for  estimating  regression-adjusted  intervention 
impacts  on  the  key  child  and  school  outcomes.  The  approach  described  here  significantly  benefited  from 
discussions  with  the  SACD  Statistical  Workgroup,  composed  of  principal  investigators  across  the  seven 
SACD  study  sites  and  representatives  of  IES  and  CDC,  in  addition  to  staff  at  MPR  and  MPR’s  subcontractor, 
University  of  Missouri-St.  Louis.  This  approach  was  used  to  choose  the  covariates  for  the  Year  1 analysis  and 
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made  use  of  the  Year  1 outcome  data.  The  covariates  that  were  selected  using  the  Year  1 data  were  also  used 
in  the  Year  2 and  Year  3 analyses  to  ensure  comparability  of  results.6 

Objectives  in  Selecting  Covariates 

Because  the  primary  goal  of  the  multiprogram  study  was  to  estimate  program  impacts,  criteria  were  adopted 
for  selecting  covariates  to  improve  the  precision  and  accuracy  of  the  impact  estimates.  The  goal  was  not  to 
build  structural  models  explaining  how  the  SACD  interventions  work  or  to  identify  mediating  variables. 
Therefore,  there  was  no  focus  on  the  signs  or  interpretations  of  the  estimated  coefficients  on  the  covariates. 

The  random  assignment  design  ensured  that  the  main  difference  between  treatment  and  control  group 
schools  at  baseline  was  the  opportunity  to  receive  SACD  program  services.  Therefore,  simple  differences  in 
mean  outcomes  between  the  treatment  and  control  groups  provided  unbiased  impact  estimates  of  the  offer  of 
SACD  program  services  on  key  outcomes.  However,  regression  models  were  used  to  estimate  the  impacts  for 
the  following  two  reasons: 

To  Increase  the  Precision  of  Impact  Estimates 

Statistical  power  was  a particularly  important  issue  for  the  SACD  evaluation  because  of  clustering  effects  due 
to  the  correlation  of  student  outcomes  within  schools.  This  was  particularly  true  at  the  site  level  due  to  the 
small  number  of  schools.  The  inclusion  of  baseline  covariates  in  the  regression  models  increased  the  precision 
of  the  impact  estimates  (that  is,  reduced  intraclass  correlations)  by  explaining  some  of  the  variation  in  mean 
outcomes  across  schools.  The  inclusion  of  school-level  covariates  (which  varied  only  between  schools), 
however,  also  resulted  in  a loss  in  degrees  of  freedom  available  for  statistical  tests,  which  reduced  statistical 
power.  This  precision  loss  was  important  at  the  site  level,  where  there  were  relatively  small  numbers  of 
schools. 

To  Adjust  for  Baseline  Differences  Between  the  Treatment  and  Control  Group  Schools 

Although  random  assignment  produced  statistically  equivalent  treatment  and  control  groups,  there  were  some 
residual  differences  in  the  average  characteristics  of  the  two  research  groups  due  to  random  sampling.  In  the 
benchmark  models,  MPR  adjusted  for  these  residual  differences  using  regression  models.  The  inclusion  of 
these  baseline  variables,  however,  affected  the  impact  estimates  only  if  they  were  also  correlated  with  the 
outcome  variables.  Thus,  as  discussed  in  the  following  section,  a covariate  with  a treatment-control  group 
difference  at  baseline  was  selected  only  if  it  also  had  some  predictive  power  in  the  regression  models. 

Tailoring  Covariates  to  Specific  Outcomes  and  Sites 

The  options  for  selecting  covariates  lie  between  picking  a set  that  is  the  same  across  all  outcomes  and  sites 
and  picking  a set  that  is  tailored  specifically  for  each  outcome  within  each  site.  While  the  former  is 
advantageous  because  of  its  parsimony,  there  is  the  risk  of  losing  precision  by  including  covariates  that  might 
not  be  correlated  with  the  particular  outcome  (due  to  reductions  in  degrees  of  freedom)  or  by  excluding 
covariates  with  predictive  power.  Completely  tailoring  sets  for  each  outcome  and  site  maximizes  precision  but 
is  computationally  intensive.  Therefore,  recognizing  that  outcomes  obtained  from  a specific  reporter  tend  to 
be  somewhat  correlated,  and  balancing  the  trade-offs  between  being  too  parsimonious  and  too  unwieldy, 
covariate  sets  were  chosen  that  differed  by  site  but  were  the  same  within  reporter  in  each  site  and  for  the  pooled 
analysis.  Thus,  the  same  set  of  covariates  was  used  to  estimate  impacts  on  child-reported  outcomes  in  a 
particular  site  (or  for  the  pooled  analysis),  but  the  set  was  allowed  to  vary  across  sites  (and  similarly  for 
outcomes  from  the  Primary  Caregiver  Report,  Teacher  Report  on  Student,  and  Teacher  Report  on  Classroom 
and  School). 


6 As  a sensitivity  analysis  for  the  use  of  the  covariates  selected  with  the  Year  1 outcome  data  in  the  analyses  of  all  3 years 
of  data,  the  covariates  were  reselected  with  the  same  approach  but  using  the  Year  2 outcome  data.  For  the  combined- 
program  analysis,  the  results  of  the  Year  1 and  Year  2 analyses  were  not  sensitive  to  which  set  of  covariates  were 
included.  There  were  several  differences  in  the  individual  program  analyses. 
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The  one  exception  to  the  rule  of  uniform  sets  within  a reporter  was  that  the  baseline  measure  of  the  outcome 
(pretest)  was  always  included  as  a covariate  in  the  regression  models.  As  noted  in  the  following  discussion, 
stepwise  regressions  found  that  the  pretests  explained  a relatively  large  proportion  of  the  variance  in  the 
outcomes. 

Implementation  of  the  Covariate  Selection  Procedure 

Table  B.3  lists  the  Year  1 (spring  2005)  outcome  variables  for  the  analysis.  The  outcomes  include  10  measures 
from  the  Child  Report,  three  measures  from  the  Primary  Caregiver  Report,  five  measures  from  the  Teacher 
Report  on  Student,  and  two  measures  from  the  Teacher  Report  on  Classroom  and  School.  These  measures 
were  constructed  from  original  scale  items  using  factor  analytic  methods. 
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Table  B.3.  Child  and  school  outcomes  used  for  covariate  selection  and  the  percentage  missing 
at  spring  2005  (Year  1)  analysis 


Outcome 

Percent 
missing 
in  combined- 
program 
sample 

Average 

percent 

missing 

across 

programs 

Minimum 

missing 

across 

programs 

Maximum 

missing 

across 

programs 

Child-reported 

Positive  School  Orientation 

4.97 

4.92 

0.64 

8.40 

Engagement  with  Learning 

4.57 

4.53 

0.64 

8.25 

Student  Afraid  at  School 

5.08 

5.04 

0.80 

8.70 

Negative  School  Orientation 

5.01 

4.97 

0.80 

8.40 

Self-Efficacy  for  Peer  Interactions 

5.06 

5.03 

0.96 

8.55 

Normative  Beliefs  About  Aggression 

5.22 

5.17 

0.80 

8.55 

Empathy 

5.10 

5.06 

0.80 

8.40 

Altruistic  Behavior 

5.08 

5.04 

0.80 

8.70 

Problem  Behavior 

5.20 

5.14 

1.28 

8.55 

Victimization  at  School 

5.06 

5.03 

1.12 

8.55 

Primary  caregiver-reported 

Positive  Social  Behavior 

22.53 

22.14 

12.96 

33.02 

Problem  Behavior 

22.51 

22.12 

12.96 

32.75 

Altruistic  Behavior 

23.79 

23.43 

14.08 

34.24 

Teacher-reported 

Positive  Social  Behavior 

1.86 

1.77 

0.15 

4.06 

Problem  Behavior 

1.70 

1.61 

0.15 

4.06 

ADHD-Related  Behavior 

1.75 

1.66 

0.15 

4.06 

Altruistic  Behavior 

4.48 

4.33 

0.87 

10.77 

Academic  Competence  and  Motivation 

1.68 

1.57 

0.16 

4.60 

Student  Support  for  Teachers 

0.59 

0.65 

0.00 

2.91 

Feelings  of  Safety 

0.59 

0.57 

0.00 

1.15 

NOTE:  ADHD  is  the  abbreviation  for  attention  deficit  hyperactivity  disorder. 
SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 


Table  B.4  lists  the  full  set  of  candidate  covariates  constructed  from  the  baseline  (fall  2004)  data  at  the  time  of 
the  Year  1 (spring  2005)  analysis.  (Note  that  because  the  first  set  of  data  was  collected  after  program 
implementation  began  at  many  sites,  these  data  are  referred  to  as  “initial  data”  rather  than  “baseline  data”  in 
the  main  body  of  this  report.)  The  set  of  potential  covariates  included  three  types  of  baseline  variables.7  First, 
the  covariates  included  child  demographic  measures  and,  from  the  Child  Report,  pretest  scores.  Second,  the 
covariates  included  primary  caregiver  characteristics  and  child  pretest  scores  from  the  Primary  Caregiver 
Report.  Finally,  the  covariates  included  child  and  primary  caregiver  involvement  pretest  scores  from  the 
Teacher  Report  on  Student. 


7 Including  covariates  to  measure  the  characteristics  of  teachers  was  considered  but  this  approach  was  rejected  for  the 
analysis  of  child  outcomes  because  teachers  were  likely  to  change  over  the  3-year  follow-up  period,  and  because  the 
SACD  interventions  might  affect  the  types  of  teachers  who  entered  and  exited  the  study  schools.  Teacher  demographic 
characteristics  were  included  as  covariates  for  the  analysis  of  the  two  teacher-reported  outcomes  from  the  TRCS. 
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Table  B.4.  Potential  covariates  and  the  percentage  missing  at  spring  2005  (Year  1)  analysis 


Percent 

missing 

Average 

percent 

Minimum 

Maximum 

in  combined- 

missing 

missing 

missing 

program 

across 

across 

across 

Potential  covariate 

sample 

programs 

programs 

programs 

Child-reported 


Female 

1.08 

1.06 

0.00 

2.25 

White  (non-Hispanic) 

9.03 

8.64 

2.56 

21.79 

Black  (non-Hispanic) 

9.03 

8.64 

2.56 

21.79 

Hispanic 

9.03 

8.64 

2.56 

21.79 

Other  ethnicity 

9.03 

8.64 

2.56 

21.79 

Age  (in  years) 

1.08 

1.06 

0.00 

2.25 

Scales 

Student  Afraid  at  School 

13.61 

13.47 

8.64 

17.19 

Altruistic  Behavior 

13.86 

13.73 

8.64 

17.05 

Empathy 

13.93 

13.80 

8.80 

17.05 

Engagement  with  Learning 

13.59 

13.44 

8.32 

17.19 

Negative  School  Orientation 

13.70 

13.56 

8.64 

17.19 

Normative  Beliefs  About  Aggression 

13.75 

13.61 

8.64 

17.05 

Positive  School  Orientation 

13.49 

13.35 

8.32 

17.19 

Problem  Behavior 

14.00 

13.86 

8.80 

17.05 

Self-Efficacy  for  Peer  Interactions 

13.82 

13.69 

8.80 

17.05 

Victimization  at  School 

14.05 

13.90 

8.80 

17.05 

Primary  caregiver-reported 

Age  (in  years) 

10.57 

10.04 

3.04 

27.47 

Completed  high  school  education  or  equivalent 

9.01 

8.62 

2.56 

22.19 

Some  college 

9.01 

8.62 

2.56 

22.19 

Bachelor’s  or  higher  degree 

9.01 

8.62 

2.56 

22.19 

Highest  level  of  education  in  household — 

Completed  high  school  or  equivalent 

9.08 

8.67 

2.56 

22.46 

Highest  level  of  education  in  household — 

Some  college 

9.08 

8.67 

2.56 

22.46 

Highest  level  of  education  in  household — 

Bachelor’s  or  higher  degree 

9.08 

8.67 

2.56 

22.46 

Mother  present  in  home  life 

8.80 

8.41 

2.56 

21.92 

Mother  and  father  present 

8.83 

8.43 

2.56 

22.06 

Respondent  someone  other  than  mother 

or  father 

8.92 

8.53 

2.56 

22.19 

Number  of  people  in  household 

9.01 

8.62 

2.56 

22.06 

Household  income:  $20,000  to  $40,000 

11.63 

11.14 

3.52 

27.20 

Household  income:  $40,000  to  $60,000 

11.63 

11.14 

3.52 

27.20 

Household  income:  More  than  $60,000 

11.63 

11.14 

3.52 

27.20 

Income-to-poverty-threshold  ratio — 

Below  135  percent 

11.61 

11.12 

3.52 

27.33 

Income-to-poverty-threshold  ratio — 

Between  135  and  185  percent 

11.61 

11.12 

3.52 

27.33 

Income-to-poverty-threshold  ratio — 

Above  1 85  percent 

11.61 

11.12 

3.52 

27.33 

See  notes  at  end  of  table. 


B-12 


Appendix  B:  Technical  Notes 


Table  B.4.  Potential  covariates  and  the  percentage  missing  at  spring  2005  (Year  1)  analysis — 
Continued 


Potential  covariate 

Percent 
missing 
in  combined- 
program 
sample 

Average 

percent 

missing 

across 

programs 

Minimum 

missing 

across 

programs 

Maximum 

missing 

across 

programs 

Full-time  employment 

9.13 

8.73 

2.72 

22.60 

Part-time  employment 

9.13 

8.73 

2.72 

22.60 

Parental  scales 

APQ-Poor  Monitoring  and  Supervision  Subscale 

18.94 

18.15 

8.32 

39.78 

APQ-Positive  Parenting  Subscale 

18.97 

18.18 

8.48 

39.51 

Child-Centered  Social  Control 

18.87 

18.08 

8.32 

39.51 

Confusion,  Hubbub,  and  Order 

19.24 

18.45 

8.80 

39.78 

Community  Resources 

19.52 

18.70 

9.44 

40.60 

Community  Risk 

20.99 

20.19 

9.60 

43.30 

Parent  and  Teacher  Involvement 

21.10 

20.41 

11.36 

42.49 

Child  scales 

Altruistic  Behavior 

18.90 

18.08 

8.48 

39.51 

Positive  Social  Behavior 

18.69 

17.88 

8.32 

39.11 

Problem  Behavior 

19.29 

18.52 

8.80 

39.24 

Teacher-reported 
Child  scales 

Academic  Competence  and  Motivation 

11.33 

10.95 

5.28 

19.76 

ADHD-Related  Behavior 

11.33 

10.96 

5.12 

19.22 

Altruistic  Behavior 

12.55 

12.13 

6.24 

21.52 

Positive  Social  Behavior 

11.49 

11.12 

5.60 

19.49 

Problem  Behavior 

11.33 

10.96 

5.12 

19.22 

Parent  involvement 

Parent-teacher  involvement 

20.74 

20.24 

10.65 

30.31 

Teacher  characteristics  (for  school  outcomes  only) 

Teacher — Female 

0.00 

0.00 

0.00 

0.00 

Teacher — Black  (non-Hispanic) 

0.24 

0.27 

0.00 

0.97 

Teacher — Hispanic 

0.24 

0.27 

0.00 

0.97 

Teacher — Other  race 

0.24 

0.27 

0.00 

0.97 

Total  experience 

0.00 

0.00 

0.00 

0.00 

Experience  at  current  school 

0.12 

0.15 

0.00 

1.05 

Regular  certification 

0.00 

0.00 

0.00 

0.00 

Other  certification 

0.00 

0.00 

0.00 

0.00 

Teacher’s  highest  degree — Bachelor’s  degree 

0.00 

0.00 

0.00 

0.00 

NOTE:  Abbreviations  are 


ADHD:  Attention  deficit  hyperactivity  disorder 
APQ:  Alabama  Parenting  Questionnaire 
SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Handling  of  Missing  Values 

Before  selecting  covariates  for  Year  1 (spring  2005)  impact  analyses,  the  missing  values  of  potential  covariates 
and  outcomes  were  considered.  Tables  B.3  and  B.4  list  the  percentages  of  observations  with  missing  values 
for  each  spring  2005  outcome  measure  and  each  potential  covariate.  The  analysis  to  select  covariates  used  a 
base  for  the  child-related  outcomes  of  all  children  with  either  a Child  Report,  a Primary  Caregiver  Report,  or 
a Teacher  Report  on  Student  at  the  first  follow-up  (and  excluded  study  non-consenters).  This  universe  of 
children  who  had  data  on  at  least  one  of  the  three  child-level  reports  (the  Child  Report,  the  Primary  Caregiver 
Report,  or  the  Teacher  Report  on  Student)  included  4,351  observations  and  varied  from  460  to  739 
observations  across  programs.  The  base  for  the  school  outcomes  included  850  third-,  fourth-,  and  fifth-grade 
teachers  with  Teacher  Report  on  Classroom  and  School  data  and  varied  from  95  to  174  teachers  across 
programs.  Outcome  measures  were  constructed  if  about  80  percent  or  more  of  pertinent  scale  items  were 
completed.8  Thus,  missing  outcomes  pertain  to  two  types  of  cases  in  the  sample  universe:  (1)  survey 
respondents  who  completed  fewer  than  80  percent  of  scale  items,  and  (2)  survey  non-respondents. 

For  the  analysis,  missing  values  for  covariates  were  imputed  using  mean  cell  imputation  procedures,  but 
missing  values  were  not  imputed  for  outcome  measures.  The  following  steps  were  used  for  the  covariate 
imputations  in  the  child-level  data: 

• For  covariates  unlikely  to  vary  over  time,  such  as  demographic  characteristics,  data  that  became 
available  during  follow-up  data  collection  were  used  before  conducting  the  cell  mean  imputations. 
Except  for  baseline  values  of  the  scales,  these  measures  included  all  child  and  primary  caregiver 
characteristics,  such  as  the  race/ethnicity  of  a child,  the  educational  attainment  of  the  primary 
caregiver,  and  whether  the  household  income  was  within  a certain  range  of  income.  Thus,  if  fall  2004 
data  on  a covariate  that  was  unlikely  to  vary  over  time  were  unavailable,  spring  2005  data  were  used 
when  available. 

• Imputation  cells  were  defined  on  the  basis  of  school,  gender,  and  race/ ethnicity.  Sample  members 
were  allocated  to  cells  defined  by  school,  gender,  and,  race/ ethnicity  (White  non-Hispanic,  Black 
non-Hispanic,  Hispanic,  and  other).  Excluded  from  the  imputation  analysis  were  cases  with  missing 
values  for  the  variables  used  to  define  cells  for  the  cell  mean  imputations,  such  as  cases  for  which  a 
child’s  gender  or  race/ ethnicity  was  unknown. 

• Imputations  for  cases  with  missing  covariates  in  a particular  cell  were  obtained  using  mean  values  for 
cases  with  nonmissing  covariate  values  in  the  same  cell.  Where  cases  with  missing  covariates  also  had 
missing  values  for  the  data  that  defined  the  cells  (such  as  school,  race/ ethnicity,  or  sex),  the 
imputations  were  performed  within  broader  cells  defined  by  the  nonmissing  characteristics.  In 
addition,  if  a cell  size  was  below  five,  the  definition  of  the  cell  was  sequentially  broadened  until  a cell 
size  of  at  least  five  was  obtained. 

• Race /ethnicity  was  imputed  using  school  and  gender.  After  imputations  on  other  covariates  were 
performed,  cases  with  missing  values  for  race/ ethnicity  were  imputed  using  mean  cell  imputations, 
where  cells  were  defined  by  school  and  gender. 

For  teacher  covariates  used  in  estimating  impacts  on  Teacher  Report  on  Classroom  and  School  school-level 
outcomes,  a similar  procedure  was  followed: 

• For  all  covariates,  the  fall  2004  data  were  used  if  available. 

• If  fall  2004  data  were  missing,  the  spring  2005  data  were  used. 


8 The  specific  cutoff  that  was  used  for  the  construction  of  outcome  measures  depended  on  the  number  of  scale  items. 
For  example,  a cutoff  of  75  percent  (3  out  of  4)  was  used  when  the  outcome  was  based  on  four  scale  items. 
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• If  a value  for  the  covariate  was  still  missing,  imputation  was  based  on  the  mean  value  for  all  teachers 
in  the  school. 

Filling  in  missing  values  of  baseline  data  had  two  closely  related  benefits.  First,  the  selection  of  covariates  for 
the  Year  1 impact  analyses  could  be  based  on  as  many  records  as  possible.  After  missing  baseline  covariate 
data  were  filled  in  through  the  use  of  follow-up  data  or  cell  mean  imputation,  the  covariates  used  in  the 
benchmark  models  were  selected  through  a stepwise  procedure  that  used  Year  1 outcomes  as  dependent 
variables  (see  Stepwise  Regression  Procedures  later  in  this  appendix).  Second,  and  similarly,  after  covariates 
were  selected,  the  estimation  procedure  used  to  generate  impact  results  could  be  based  on  as  many  records  as 
possible.  In  this  way,  estimating  the  results  was  not  hampered  by  missing  initial  data. 

Filling  in  missing  values  for  baseline  data  for  the  Year  2 and  Year  3 itnpact  analyses.  After  the  benchmark  model 
covariates  were  selected  and  the  Year  1 analyses  were  conducted,  additional  follow-up  data  became  available 
for  the  2005-06  and  2006-07  school  years.  Data  from  these  follow-up  periods  were  used  to  fill  in  missing 
baseline  covariate  data  for  new  students  who  entered  the  study  schools  after  the  spring  2005  data  collection 
and  for  students  who  had  been  in  the  study  schools  from  the  start  of  the  study  (thus  avoiding  the  need  to  use 
cell  mean  imputation).  Tables  B.5  and  B.6  show  the  percentages  of  observations  with  missing  values  for  each 
spring  2006  (Year  2)  outcome  measure  and  each  potential  covariate,  and  tables  B.7  and  B.8  show  the 
percentages  of  observations  with  missing  values  for  each  spring  2007  (Year  3)  outcome  measure  and  each 
potential  covariate. 

The  procedure  to  fill  in  missing  data  for  the  covariates  used  in  the  Year  2 (spring  2006)  and  the  Year  3 (spring 
2007)  regression  analyses  was  similar  to  that  used  for  the  Year  1 (spring  2005)  analyses.  If  fall  2004  initial  data 
were  available,  then  those  data  were  used.  If  not,  then  spring  2005  data  on  covariates,  such  as  the 
race/ ethnicity  of  the  child  and  the  educational  attainment  of  the  primary  caregiver,  were  used  if  possible  (that 
is,  if  the  spring  2005  data  item  was  available  and  if  the  data  item  was  unlikely  to  vary  over  time).  If  fall  2004 
data  and  spring  2005  data  were  unavailable,  then  fall  2005  data  were  used  to  fill  in  the  baseline  data.  Next, 
spring  2006  data  were  used.  Finally,  spring  2007  data  were  used.  If  the  information  was  unavailable  from  all 
these  data  collection  efforts,  or  if  the  data  item  was  likely  to  vary  over  time  (as  is  the  case  with  pretest  data), 
the  cell  mean  imputation  process  was  used  to  fill  in  the  missing  data. 
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Table  B.5.  Child  and  school  outcomes  used  for  covariate  selection  and  the  percentage  missing 
at  spring  2006  (Year  2)  analysis 


Percent 

missing 

Average 

percent 

Minimum 

Maximum 

in  combined- 

missing 

missing 

missing 

program 

across 

across 

across 

Outcome 

sample 

programs 

programs 

programs 

Child-reported 


Positive  School  Orientation 

5.48 

5.43 

1.17 

9.22 

Engagement  with  Learning 

5.10 

5.06 

0.73 

8.32 

Student  Afraid  at  School 

5.50 

5.43 

1.03 

8.15 

Negative  School  Orientation 

5.50 

5.44 

0.73 

9.04 

Self-Efficacy  for  Peer  Interactions 

5.64 

5.59 

1.17 

9.04 

Normative  Beliefs  About  Aggression 

5.85 

5.77 

1.32 

9.38 

Empathy 

5.74 

5.66 

0.88 

9.65 

Altruistic  Behavior 

6.04 

5.95 

1.32 

9.65 

Problem  Behavior 

6.02 

5.93 

1.32 

9.51 

Victimization  at  School 

6.09 

6.00 

1.47 

9.65 

Primary  caregiver-reported 

Positive  Social  Behavior 

25.47 

25.24 

13.49 

38.18 

Problem  Behavior 

25.47 

25.24 

13.49 

38.18 

Altruistic  Behavior 

26.43 

26.20 

15.69 

39.27 

Teacher-reported 

Positive  Social  Behavior 

0.87 

0.82 

0.00 

1.63 

Problem  Behavior 

0.89 

0.85 

0.00 

1.63 

ADHD-Related  Behavior 

0.84 

0.80 

0.00 

1.63 

Altruistic  Behavior 

2.29 

2.18 

0.16 

4.51 

Academic  Competence  and  Motivation 

1.03 

0.95 

0.00 

2.17 

Student  Support  for  Teachers 

0.36 

0.33 

0.00 

0.89 

Feelings  of  Safety 

0.36 

0.37 

0.00 

0.89 

NOTE:  ADHD  is  the  abbreviation  for  attention  deficit  hyperactivity  disorder. 
SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Table  B.6.  Potential  covariates  and  the  percentage  missing  at  spring  2006  (Year  2)  analysis 


Percent 

missing 

Average 

percent 

Minimum 

Maximum 

in  combined- 

missing 

missing 

missing 

program 

across 

across 

across 

Potential  covariate 

sample 

programs 

programs 

programs 

Child-reported 


Female 

0.47 

0.48 

0.00 

0.90 

White  (non-Hispanic) 

6.53 

6.26 

2.20 

15.35 

Black  (non-Hispanic) 

6.53 

6.26 

2.20 

15.35 

Hispanic 

6.53 

6.26 

2.20 

15.35 

Other  ethnicity 

6.53 

6.26 

2.20 

15.35 

Age  (in  years) 

0.23 

0.24 

0.00 

0.54 

Scales 

Student  Afraid  at  School 

27.57 

27.54 

23.61 

34.36 

Altruistic  Behavior 

27.76 

27.74 

23.90 

34.72 

Empathy 

27.79 

27.76 

24.05 

34.72 

Engagement  with  Learning 

27.55 

27.53 

23.31 

34.54 

Negative  School  Orientation 

27.62 

27.59 

23.76 

34.36 

Normative  Beliefs  About  Aggression 

27.65 

27.62 

23.90 

34.36 

Positive  School  Orientation 

27.48 

27.46 

23.46 

34.36 

Problem  Behavior 

27.81 

27.78 

24.05 

34.72 

Self-Efficacy  for  Peer  Interactions 

27.69 

27.66 

24.05 

34.36 

Victimization  at  School 

27.88 

27.85 

24.05 

34.90 

Primary  caregiver-reported 

Age  (in  years) 

6.84 

6.53 

2.64 

16.30 

Completed  high  school  education  or  equivalent 

6.46 

6.18 

2.05 

15.49 

Some  college 

6.46 

6.18 

2.05 

15.49 

Bachelor’s  or  higher  degree 

6.46 

6.18 

2.05 

15.49 

Highest  level  of  education  in  household — 

Completed  high  school  or  equivalent 

6.44 

6.17 

2.05 

15.35 

Highest  level  of  education  in  household — 

Some  college 

6.44 

6.17 

2.05 

15.35 

Highest  level  of  education  in  household — 

Bachelor’s  or  higher  degree 

6.44 

6.17 

2.05 

15.35 

Mother  present  in  home  life 

6.44 

6.16 

2.05 

15.49 

Mother  and  father  present 

6.44 

6.16 

2.05 

15.49 

Respondent  someone  other  than  mother 

or  father 

6.48 

6.20 

2.05 

15.76 

Number  of  people  in  household 

6.55 

6.29 

2.35 

15.49 

Household  income:  $20,000  to  $40,000 

8.12 

7.77 

2.79 

19.43 

Household  income:  $40,000  to  $60,000 

8.12 

7.77 

2.79 

19.43 

Household  income:  More  than  $60,000 

8.12 

7.77 

2.79 

19.43 

Income-to-poverty-threshold  ratio — 

Below  135  percent 

8.22 

7.86 

3.08 

19.57 

Income-to-poverty-threshold  ratio — 

Between  135  and  185  percent 

8.22 

7.86 

3.08 

19.57 

See  notes  at  end  of  table. 
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Table  B.6.  Potential  covariates  and  the  percentage  missing  at  spring  2006  (Year  2)  analysis — 
Continued 


Potential  covariate 

Percent 
missing 
in  combined- 
program 
sample 

Average 

percent 

missing 

across 

programs 

Minimum 

missing 

across 

programs 

Maximum 

missing 

across 

programs 

Income-to-poverty-threshold  ratio — 

Above  1 85  percent 

8.22 

7.86 

3.08 

19.57 

Full-time  employment 

6.55 

6.27 

2.05 

15.76 

Part-time  employment 

6.55 

6.27 

2.05 

15.76 

Parental  scales 

APQ — Poor  Monitoring  and  Supervision 

Subscale 

31.60 

30.99 

23.02 

50.68 

APQ — Positive  Parenting  Subscale 

31.65 

31.04 

23.17 

50.41 

Child-Centered  Social  Control 

31.44 

30.85 

22.73 

50.14 

Confusion,  Hubbub,  and  Order 

31.84 

31.24 

23.46 

50.27 

Community  Resources 

31.93 

31.31 

24.00 

50.68 

Community  Risk 

32.89 

32.27 

23.90 

52.58 

Parent  and  teacher  Involvement 

33.47 

32.92 

25.37 

52.85 

Child  scales 

Altruistic  Behavior 

31.53 

30.91 

23.17 

50.54 

Positive  Social  Behavior 

31.32 

30.70 

23.02 

50.00 

Problem  Behavior 

31.91 

31.30 

23.61 

50.41 

Teacher-reported 
Child  scales 

Academic  Competence  and  Motivation 

25.63 

25.39 

19.33 

34.54 

ADHD-Related  Behavior 

25.61 

25.38 

19.11 

34.72 

Altruistic  Behavior 

26.66 

26.40 

19.11 

37.43 

Positive  Social  Behavior 

25.77 

25.53 

19.11 

34.72 

Problem  Behavior 

25.61 

25.38 

19.11 

34.72 

Parent  involvement 

Parent  and  teacher  involvement 

33.47 

33.06 

22.44 

43.34 

Teacher  characteristics  (for  school 
outcomes  only) 

Teacher — Female 

0.00 

0.00 

0.00 

0.00 

Teacher — Black  (non-Hispanic) 

0.48 

0.41 

0.00 

1.16 

Teacher — Hispanic 

0.48 

0.41 

0.00 

1.16 

Teacher — Other  race 

0.48 

0.41 

0.00 

1.16 

Total  experience 

0.24 

0.30 

0.00 

1.22 

Experience  at  current  school 

0.24 

0.30 

0.00 

1.22 

Regular  certification 

0.00 

0.00 

0.00 

0.00 

Other  certification 

0.00 

0.00 

0.00 

0.00 

Teacher’s  highest  degree — 

Bachelor’s  degree 

0.00 

0.00 

0.00 

0.00 

NOTE:  Abbreviations  are 


ADHD:  Attention  deficit  hyperactivity  disorder 
APQ:  Alabama  Parenting  Questionnaire 
SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Table  B.7.  Child  and  school  outcomes  used  for  covariate  selection  and  the  percentage 
missing  at  spring  2007  (Year  3)  analysis 


Outcome 

Percent 
missing 
in  combined- 
program 
sample 

Average 

percent 

missing 

across 

programs 

Minimum 

missing 

across 

programs 

Maximum 

missing 

across 

programs 

Child-reported 

Positive  School  Orientation 

4.33 

4.34 

0.44 

9.79 

Engagement  with  Learning 

4.07 

4.06 

0.74 

9.79 

Student  Afraid  at  School 

4.43 

4.44 

0.30 

9.95 

Negative  School  Orientation 

4.33 

4.35 

0.74 

9.95 

Self-Efficacy  for  Peer  Interactions 

4.41 

4.42 

0.59 

9.79 

Normative  Beliefs  About  Aggression 

4.36 

4.35 

0.74 

9.95 

Empathy 

4.41 

4.41 

0.44 

9.79 

Altruistic  Behavior 

4.53 

4.53 

0.74 

9.79 

Problem  Behavior 

4.62 

4.63 

0.44 

9.79 

Victimization  at  School 

4.65 

4.64 

0.89 

9.95 

Primary  caregiver-reported 

Positive  Social  Behavior 

31.28 

31.04 

16.89 

48.10 

Problem  Behavior 

31.25 

31.02 

16.89 

47.96 

Altruistic  Behavior 

32.22 

31.98 

18.07 

49.56 

Teacher-reported 

Positive  Social  Behavior 

2.00 

1.99 

0.58 

6.20 

Problem  Behavior 

1.90 

1.89 

0.44 

6.04 

ADHD-Related  Behavior 

1.97 

1.96 

0.58 

6.36 

Altruistic  Behavior 

3.83 

4.04 

0.73 

11.95 

Academic  Competence  and  Motivation 

1.88 

1.84 

0.15 

5.88 

Student  Support  for  Teachers 

0.61 

0.63 

0.00 

1.90 

Feelings  of  Safety 

0.85 

0.83 

0.00 

2.40 

NOTE:  ADHD  is  the  abbreviation  for  attention  deficit  hyperactivity  disorder. 
SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Table  B.8.  Potential  covariates  and  the  percentage  missing  at  spring  2007  (Year  3)  analysis 


Potential  covariate 

Percent 
missing 
in  combined- 
program 
sample 

Average 

percent 

missing 

across 

programs 

Minimum 

missing 

across 

programs 

Maximum 

missing 

across 

programs 

Child-reported 

Female 

0.60 

0.61 

0.15 

1.47 

White  (non-Hispanic) 

7.44 

7.32 

1.93 

17.49 

Black  (non-Hispanic) 

7.44 

7.32 

1.93 

17.49 

Hispanic 

7.44 

7.32 

1.93 

17.49 

Other  ethnicity 

7.44 

7.32 

1.93 

17.49 

Age  (in  years) 

12.91 

13.14 

6.81 

19.85 

Scales 

Student  Afraid  at  School 

34.48 

34.68 

27.41 

43.55 

Altruistic  Behavior 

34.67 

34.88 

27.26 

43.74 

Empathy 

34.70 

34.90 

27.41 

43.74 

Engagement  with  Learning 

34.43 

34.64 

27.11 

43.74 

Negative  School  Orientation 

34.53 

34.72 

27.41 

43.55 

Normative  Beliefs  About  Aggression 

34.58 

34.78 

27.26 

43.55 

Positive  School  Orientation 

34.41 

34.61 

27.11 

43.55 

Problem  Behavior 

34.70 

34.90 

27.41 

43.74 

Self-Efficacy  for  Peer  Interactions 

34.63 

34.83 

27.41 

43.55 

Victimization  at  School 

34.79 

34.99 

27.41 

43.93 

Primary  caregiver-reported 
Age  (in  years) 

7.95 

7.82 

1.93 

18.51 

Completed  high  school  education  or  equivalent 

7.39 

7.27 

1.93 

17.64 

Some  college 

7.39 

7.27 

1.93 

17.64 

Bachelor’s  or  higher  degree 

7.39 

7.27 

1.93 

17.64 

Highest  level  of  education  in  household — 
Completed  high  school  or  equivalent 

7.44 

7.31 

2.07 

17.49 

Highest  level  of  education  in  household — 
Some  college 

7.44 

7.31 

2.07 

17.49 

Highest  level  of  education  in  household — 
Bachelor’s  or  higher  degree 

7.44 

7.31 

2.07 

17.49 

Mother  present  in  home  life 

7.37 

7.24 

1.93 

17.64 

Mother  and  father  present 

7.37 

7.24 

1.93 

17.64 

Respondent  someone  other  than  mother 
or  father 

7.44 

7.31 

1.93 

17.64 

Number  of  people  in  household 

7.56 

7.45 

2.07 

17.49 

Household  income:  $20,000  to  $40,000 

9.03 

8.87 

2.67 

21.28 

Household  income:  $40,000  to  $60,000 

9.03 

8.87 

2.67 

21.28 

Household  income:  More  than  $60,000 

9.03 

8.87 

2.67 

21.28 

Income-to-poverty-threshold  ratio — 
Below  135  percent 

9.15 

9.00 

2.67 

21.28 

Income-to-poverty-threshold  ratio — 
Between  135  and  185  percent 

9.15 

9.00 

2.67 

21.28 

Income-to-poverty-threshold  ratio — 
Above  1 85  percent 

9.15 

9.00 

2.67 

21.28 

Full-time  employment 

7.49 

7.36 

1.93 

17.78 

Part-time  employment 

7.49 

7.36 

1.93 

17.78 

See  notes  at  end  of  table. 
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Table  B.8.  Potential  covariates  and  the  percentage  missing  at  spring  2007  (Year  3)  analysis — 
Continued 


Potential  covariate 

Percent 
missing 
in  combined- 
program 
sample 

Average 

percent 

missing 

across 

programs 

Minimum 

missing 

across 

programs 

Maximum 
missing  across 
programs 

Parental  scales 

APQ — Poor  Monitoring  and  Supervision 

Subscale 

38.02 

37.73 

26.67 

55.69 

APQ — Positive  Parenting  Subscale 

38.07 

37.79 

26.81 

55.54 

Child-Centered  Social  Control 

37.92 

37.65 

26.37 

55.25 

Confusion,  Hubbub,  and  Order 

38.29 

38.01 

27.11 

55.25 

Community  Resources 

38.43 

38.13 

27.56 

55.83 

Community  Risk 

39.25 

38.94 

27.41 

57.73 

Parent  and  Teacher  Involvement 

39.66 

39.45 

28.59 

57.43 

Child  scales 

Altruistic  Behavior 

37.92 

37.62 

26.81 

55.69 

Positive  Social  Behavior 

37.76 

37.46 

26.67 

55.10 

Problem  Behavior 

38.31 

38.03 

27.26 

55.25 

Teacher-reported 
Child  scales 

Academic  Competence  and  Motivation 

32.65 

32.67 

24.89 

42.77 

ADHD-Related  Behavior 

32.65 

32.69 

24.89 

43.16 

Altruistic  Behavior 

33.57 

33.58 

25.93 

45.47 

Positive  Social  Behavior 

32.80 

32.82 

25.33 

43.16 

Problem  Behavior 

32.65 

32.69 

24.89 

43.16 

Parent  involvement 

Parent  and  teacher  involvement 

39.73 

39.51 

29.25 

49.52 

Teacher  characteristics  (for  school 
outcomes  only) 

Teacher — Female 

0.24 

0.23 

0.00 

1.60 

Teacher — Black  (non-Hispanic) 

0.49 

0.45 

0.00 

1.60 

Teacher — Hispanic 

0.49 

0.45 

0.00 

1.60 

Teacher — Other  race 

0.49 

0.45 

0.00 

1.60 

Total  experience 

0.37 

0.32 

0.00 

1.60 

Experience  at  current  school 

0.49 

0.45 

0.00 

1.60 

Regular  certification 

0.00 

0.00 

0.00 

0.00 

Other  certification 

0.00 

0.00 

0.00 

0.00 

Teacher’s  highest  degree — 

Bachelor’s  degree 

0.24 

0.23 

0.00 

1.60 

NOTE:  Abbreviations  are 


ADHD:  Attention  deficit  hyperactivity  disorder 
APQ:  Alabama  Parenting  Questionnaire 
SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Stepwise  Regression  Procedures 

For  the  selection  of  covariates  to  be  used  in  the  benchmark  impact  models,  stepwise  regression  methods  were 
used  to  identify  potential  covariates  with  significant  explanatory  power.9  The  treatment  binary  variable 
(measuring  estimated  impacts);  program-specific  binary  variables  (for  the  pooled  models);  and  the  age, 
gender,  and  race/ethnicity  of  the  child  were  included  (these  covariates  were  separately  tested  as  to  whether 
they  would  be  selected  under  the  implementation  rules).  At  each  step  of  the  stepwise  procedure,  the  variable 
with  the  smallest  /(-value  below  a preset  threshold  level  was  included  in  the  model  while  variables  already 
selected  were  evaluated  to  see  if  any  could  be  removed;  the  variable  with  a /(-value  greater  than  the  critical  p- 
value  of  0.32  and  whose  removal  would  least  lower  the  adjusted  R2  value  was  removed.  The  critical  /(-value 
was  set  at  0.32  to  correspond  to  a /-statistic  of  1,  which  is  the  smallest  value  of  the  /-statistic  at  which  the 
addition  of  a variable  in  a model  increases  the  adjusted  R 2 value.10 

Rules  for  Selecting  Covariates  Using  Year  1 Data 

The  rules  for  selecting  covariates  had  two  key  features:  (1)  they  identified  covariates  that  had  a treatment- 
control  group  difference  at  baseline  and  that  showed  some  explanatory  power  in  the  regression  models,  and 
(2)  they  identified  covariates  that  had  no  treatment-control  group  difference  at  baseline  and  that  had 
substantial  explanatory  power  in  the  regression  models  for  a majority  of  outcomes.  Specifically,  based  on 
regressions  using  spring  2005  (Year  1)  outcomes,  a unique  set  of  covariates  by  reporter  and  program  were 
selected  using  sequential  rules.  First,  a variable  was  included  that  had  a statistically  significant  treatment- 
control  group  baseline  difference  in  the  variable,  and  that  was  selected  in  the  final  stepwise  model  for  at  least 
one  of  the  outcomes  for  a given  reporter.  Second,  a variable  with  no  treatment-control  group  difference  at 
baseline  was  included  if  it  was  selected  by  the  stepwise  regressions  for  about  60  percent  or  more  of  the 
outcomes.  Third,  covariates  were  included  to  increase  face  validity.  That  is,  variables  that  measured  age, 
gender,  race,  and  ethnicity  of  the  child  (or  of  the  teacher  for  Teacher  Report  on  Classroom  and  School 
outcomes)  were  included  if  they  had  not  already  been  selected.  Finally,  for  completeness,  covariates  were 
selected  if  they  completed  a set  of  categorical  variables  if  one  of  the  categories  had  been  previously  selected. 

Tables  B.9  through  B.12  list  the  sets  of  covariates  selected  as  a result  of  these  rules.  The  covariates  are  listed 
separately  for  the  combined-program  analysis  and  the  program-specific  analyses.  Table  B.9  provides  details 
for  all  child -reported  outcomes,  table  B.10  provides  details  for  all  primary  caregiver-reported  outcomes,  table 
B.ll  does  the  same  for  all  teacher-reported  child  outcomes,  and  table  B.12  does  so  for  all  teacher -reported 
school  outcomes. 

Key  features  of  the  selected  covariates  are  as  follows: 

• While  baseline  values  of  the  scale  under  consideration  were  always  selected  as  covariates,  many  scales 
were  also  selected  across  outcomes  and  reporters.  For  example,  7 of  the  10  child-reported  scales  and 
2 of  the  6 teacher-reported  scales  were  selected  as  covariates  for  child-reported  outcomes  in  the 
combined -program  sample  (table  B.9).  Similarly,  for  primary  caregiver-reported  outcomes,  5 of  the 
10  child-reported  scales,  2 of  the  3 primary  caregiver-reported  scales,  and  3 of  the  6 teacher-reported 
scales  were  selected  as  covariates  in  the  combined -program  sample  (table  B.10).  Finally,  at  the 
combined-program  level,  3 of  the  10  child -reported  scales,  all  3 primary  caregiver-reported  scales, 
and  4 of  the  6 teacher-reported  scales  were  selected  as  covariates  for  the  teacher-reported  child-level 
outcomes  (table  B.l  1). 


9 The  PROC  REG  procedure  in  SAS  was  used  with  the  SELECTION  option  set  to  STEPWISE. 

10  The  stepwise  procedure  in  SAS  did  not  adjust  for  clustering  effects.  Thus,  after  selecting  covariates  from  the  stepwise 
procedure,  these  covariates  were  used  to  re-estimate  the  models  in  SAS  PROC  MIXED  (which  accounts  for  school-level 
clustering  effects)  and  the  final  covariate  sets  were  refined  accordingly.  Because  of  small  clustering  effects,  however, 
these  refinements  were  minor. 


B-22 


Appendix  B:  Technical  Notes 


• Most  demographic  and  socioeconomic  characteristics  of  the  primary  caregiver  were  selected  as 
covariates  for  primary  caregiver-reported  outcomes.  However,  these  measures  were  selected  less 
consistendy  for  the  child-  and  teacher-reported  outcomes. 

• Considerable  variation  existed  in  the  selected  covariates  across  programs.  This  result  likely  reflected 
differences  in  study  populations  across  programs. 

• The  baseline  value  of  the  outcome  under  consideration  typically  explained  the  highest  proportion  of 
variance  in  the  outcome  among  all  covariates.  Using  spring  2005  data,  tables  B.13  and  B.14  compare 
the  adjusted  R2  values  of  the  models  that  included  only  the  corresponding  baseline  pretest  value  with 
models  that  included  the  final  selected  set  of  covariates.  Results  are  reported  for  models  run  at  the 
child  and  school  levels.  The  tables  indicate  that  R-  values  did  not  increase  substantially  when 
covariates  other  than  the  pretests  were  included  in  the  models.  For  Year  2 (spring  2006  data),  tables 
B.15  and  B.16  show  similar  information  and  results,  and  tables  B.17  and  B.18  show  this  information 
for  Year  3 (spring  2007  data). 
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Table  B.9.  Covariates  selected  for  child-reported  outcomes,  for  the  combined-program  sample 
and  program-specific  samples 


Combined- 

program 


Potential  covariate 

sample 

ABC 

PA 

SS 

4Rs 

CSP 

PATHS 

LBW 

Total  number  of  covariates 

26 

11 

20 

11 

10 

17 

18 

6 

Child-reported 

Female 

✓ 

✓ 

✓ 

✓ 

✓ 

✓ 

V 

Black  (non-Hispanic) 

✓ 

✓ 

✓ 

✓ 

✓ 

✓ 

✓ 

V 

Hispanic 

✓ 

✓ 

✓ 

✓ 

✓ 

✓ 

V 

Other  ethnicity 

✓ 

✓ 

✓ 

✓ 

✓ 

✓ 

✓ 

V 

Age  (in  years) 

✓ 

✓ 

✓ 

✓ 

✓ 

✓ 

✓ 

V 

Scales 

Student  Afraid  at  School 

Altruistic  Behavior 

✓ 

✓ 

Empathy 

✓ 

Engagement  with  Learning 

✓ 

✓ 

Negative  School  Orientation 

✓ 

✓ 

✓ 

Normative  Beliefs  About  Aggression 

✓ 

✓ 

Positive  School  Orientation 

✓ 

Problem  Behavior 

✓ 

Self-Efficacy  for  Peer  Interactions 

✓ 

✓ 

Victimization  at  School 

✓ 

✓ 

Primary  caregiver-reported 

Age  (in  years) 

✓ 

Completed  high  school 
education  or  equivalent 

✓ 

✓ 

Some  college 

✓ 

✓ 

Bachelor’s  or  higher  degree 

✓ 

Highest  level  of  education  in  household — 
Completed  high  school  or  equivalent 

✓ 

✓ 

✓ 

Highest  level  of  education  in  household — 
Some  college 

V 

✓ 

✓ 

Highest  level  of  education  in  household — 
Bachelor’s  or  higher  degree 

V 

✓ 

V 

Mother  present  in  home  life 

Mother  and  father  present 

V 

Respondent  someone  other  than  mother 
or  father 

✓ 

Number  of  people  in  household 

✓ 

✓ 

✓ 

Household  income:  $20,000  to  $40,000 

✓ 

✓ 

Household  income:  $40,000  to  $60,000 

V 

✓ 

Household  income:  More  than  $60,000 

✓ 

✓ 

See  notes  at  end  of  table. 
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Table  B.9.  Covariates  selected  for  child-reported  outcomes,  for  the  combined-program  sample 
and  program-specific  samples — Continued 


Combined- 

program 


Potential  covariate 

sample 

ABC 

PA 

SS 

4Rs 

CSP 

PATHS 

LBW 

Income-to-poverty-threshold  ratio — 
Below  135  percent 

✓ 

Income-to-poverty-threshold  ratio — 
Between  135  and  185  percent 

✓ 

Full-time  employment 

✓ 

V 

Part-time  employment 

✓ 

V 

Parental  scales 

APQ — Poor  Monitoring  and  Supervision 
Subscale 

✓ 

V 

APQ — Positive  Parenting  Subscale 

Child-Centered  Social  Control 

Confusion,  Hubbub,  and  Order 

✓ 

Community  Resources 

Community  Risk 

✓ 

Parent  and  Teacher  Involvement 

Child  scales 

Altruistic  Behavior 

✓ 

Positive  Social  Behavior 

✓ 

✓ 

Problem  Behavior 

✓ 

Teacher-reported 

Child  scales 

Academic  Competence  and  Motivation 

✓ 

ADHD-Related  Behavior 

✓ 

✓ 

✓ 

Altruistic  Behavior 

Positive  Social  Behavior 

✓ 

Problem  Behavior 

Parent  and  Teacher  Involvement 

✓ 

NOTE:  Abbreviations  are 

ABC:  Academic  and  Behavioral  Competencies  P rogram 
CSP:  Competence  Support  Program 
LBW:  Love  In  a Big  World 
PA:  Positive  Action 

PATHS:  Promoting  Alternative  Thinking  Strategies 

4Rs:  The  4Rs  Program  (Reading,  Writing,  Respect,  and  Resolution) 

SS:  Second  Step 

ADHD:  Attention  deficit  hyperactivity  disorder 
APQ:  Alabama  Parenting  Questionnaire 
■S : Covariate  used 
Blank  cell:  Covariate  not  used 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Table  B.10.  Covariates  selected  for  primary  caregiver-reported  child  outcomes,  for  the 
combined-program  and  program-specific  samples 


Combined- 

program 


Potential  covariate 

sample 

ABC 

PA 

ss 

4Rs 

CSP 

PATHS 

LBW 

Total  number 

35 

13 

26 

28 

26 

24 

27 

25 

Child-reported 

Female 

✓ 

V 

✓ 

V 

V 

Black  (non-Hispanic) 

✓ 

V 

✓ 

V 

V 

Hispanic 

✓ 

V 

✓ 

V 

V 

Other  ethnicity 

✓ 

V 

✓ 

V 

V 

Age  (in  years) 

✓ 

V 

s 

V 

V 

Scales 

Student  Afraid  at  School 

V 

Altruistic  Behavior 

Empathy 

V 

V 

Engagement  with  Learning 

✓ 

V 

Negative  School  Orientation 

✓ 

V 

✓ 

✓ 

V 

Normative  Beliefs  About  Aggression 

✓ 

V 

V 

Positive  School  Orientation 

✓ 

V 

Problem  Behavior 

✓ 

✓ 

✓ 

Self-Efficacy  for  Peer  Interactions 

✓ 

✓ 

V 

Victimization  at  School 

V 

✓ 

Primary  caregiver-reported 

Age  (in  years) 

✓ 

V 

✓ 

✓ 

Completed  high  school  education  or  equivalent 

✓ 

V 

V 

✓ 

✓ 

Some  college 

✓ 

V 

V 

✓ 

Bachelor’s  or  higher  degree 

✓ 

V 

V 

✓ 

Highest  level  of  education  in  household — 
Completed  high  school  or  equivalent 

✓ 

V 

Highest  level  of  education  in  household — 
Some  college 

✓ 

V 

✓ 

✓ 

Highest  level  of  education  in  household — 
Bachelor’s  or  higher  degree 

✓ 

s 

✓ 

V 

Mother  present  in  home  life 

✓ 

V 

Mother  and  father  present 

✓ 

V 

✓ 

Respondent  someone  other  than  mother 
or  father 

✓ 

V 

✓ 

V 

Number  of  people  in  household 

✓ 

V 

s 

Household  income:  $20,000  to  $40,000 

✓ 

V 

V 

V 

Household  income:  $40,000  to  $60,000 

✓ 

V 

V 

V 

Household  income:  More  than  $60,000 

✓ 

V 

V 

V 

Income-to-poverty-threshold  ratio — 
Below  135  percent 

✓ 

V 

V 

V 

Income-to-poverty-threshold  ratio — 
Between  135  and  185  percent 

✓ 

V 

✓ 

V 

V 

Full-time  employment 

✓ 

V 

✓ 

V 

V 

Part-time  employment 

✓ 

V 

✓ 

V 

V 

See  notes  at  end  of  table. 
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Table  B.10.  Covariates  selected  for  primary  caregiver-reported  child  outcomes,  for  the 
combined-program  and  program-specific  samples — Continued 


Combined- 

program 


Potential  covariate 

sample 

ABC 

PA 

SS 

4Rs 

CSP 

PATHS 

LBW 

Parental  scales 

APQ — Poor  Monitoring  and  Supervision 
Subscale 

✓ 

✓ 

✓ 

APQ — Positive  Parenting  Subscale 

V 

✓ 

Child-Centered  Social  Control 

✓ 

Confusion,  Hubbub,  and  Order 

✓ 

✓ 

✓ 

✓ 

Community  Resources 

✓ 

Community  Risk 

✓ 

✓ 

V 

Parent  and  Teacher  Involvement 

✓ 

Child  scales 

Altruistic  Behavior 

V 

Positive  Social  Behavior 

✓ 

V 

Problem  Behavior 

v 

V 

✓ 

V 

Teacher-reported 

Child  scales 

Academic  Competence  and  Motivation 

V 

✓ 

ADHD-Related  Behavior 

✓ 

Altruistic  Behavior 

V 

✓ 

V 

✓ 

V 

Positive  Social  Behavior 

V 

Problem  Behavior 

V 

V 

Parent  and  Teacher  Involvement 

V 

NOTE:  Abbreviations  are 

ABC:  Academic  and  Behavioral  Competencies  Program 
CSP:  Competence  Support  Program 
LBW:  Love  In  a Big  World 
PA:  Positive  Action 

PATHS:  Promoting  Alternative  Thinking  Strategies 

4Rs:  The  4Rs  Program  (Reading,  Writing,  Respect,  and  Resolution) 

SS:  Second  Step 

ADHD:  Attention  deficit  hyperactivity  disorder 
APQ:  Alabama  Parenting  Questionnaire 
■S : Covariate  used 
Blank  cell:  Covariate  not  used 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Table  B.11.  Covariates  selected  for  teacher-reported  child  outcomes,  for  the  combined-program 
and  program-specific  samples 


Combined- 

program 


Potential  covariate 

sample 

ABC 

PA 

SS 

4Rs 

CSP 

PATHS 

LBW 

Total  number  of  covariates 

28 

24 

22 

21 

20 

25 

27 

21 

Child-reported 

Female 

✓ 

✓ 

✓ 

✓ 

✓ 

✓ 

✓ 

V 

Black  (non-Hispanic) 

✓ 

✓ 

✓ 

✓ 

✓ 

✓ 

✓ 

V 

Hispanic 

✓ 

✓ 

✓ 

✓ 

✓ 

✓ 

✓ 

V 

Other  ethnicity 

✓ 

✓ 

✓ 

✓ 

✓ 

✓ 

✓ 

V 

Age  (in  years) 

✓ 

✓ 

✓ 

✓ 

✓ 

✓ 

✓ 

V 

Scales 

Student  Afraid  at  School 

✓ 

✓ 

✓ 

Altruistic  Behavior 

✓ 

✓ 

✓ 

s 

Empathy 

✓ 

✓ 

s 

Engagement  with  Learning 

✓ 

✓ 

✓ 

✓ 

Negative  School  Orientation 

✓ 

✓ 

✓ 

Normative  Beliefs  About  Aggression 

✓ 

✓ 

✓ 

Positive  School  Orientation 

✓ 

✓ 

Problem  Behavior 

✓ 

✓ 

Self-Efficacy  for  Peer  Interactions 

✓ 

V 

s 

Victimization  at  School 

✓ 

V 

Primary  caregiver-reported 

Age  (in  years) 

✓ 

✓ 

✓ 

Completed  high  school  education  or  equivalent 

✓ 

✓ 

✓ 

✓ 

Some  college 

✓ 

✓ 

✓ 

✓ 

✓ 

Bachelor’s  or  higher  degree 

✓ 

✓ 

✓ 

✓ 

✓ 

Highest  level  of  education  in  household — 
Completed  high  school  or  equivalent 

✓ 

✓ 

✓ 

Highest  level  of  education  in  household — 
Some  college 

✓ 

✓ 

Highest  level  of  education  in  household — 
Bachelor’s  or  higher  degree 

✓ 

✓ 

✓ 

Mother  present  in  home  life 

✓ 

✓ 

✓ 

V 

Mother  and  father  present 

✓ 

✓ 

✓ 

✓ 

Respondent  someone  other  than  mother 
or  father 

✓ 

✓ 

✓ 

✓ 

Number  of  people  in  household 

✓ 

✓ 

✓ 

Household  income:  $20,000  to  $40,000 

V 

✓ 

Household  income:  $40,000  to  $60,000 

✓ 

✓ 

Household  income:  More  than  $60,000 

✓ 

✓ 

Income-to-poverty-threshold  ratio — 
Below  135  percent 

✓ 

V 

Income-to-poverty-threshold  ratio — 
Between  135  and  185  percent 

✓ 

V 

Full-time  employment 

✓ 

V 

✓ 

✓ 

✓ 

V 

Part-time  employment 

✓ 

V 

✓ 

✓ 

✓ 

V 

See  notes  at  end  of  table. 
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Table  B.1 1 . Covariates  selected  for  teacher-reported  child  outcomes,  for  the  combined- 
program  and  program-specific  samples — Continued 


Combined- 

program 


Potential  covariate 

sample 

ABC 

PA 

SS 

4Rs 

CSP 

PATHS 

LBW 

Parental  scales 

APQ — Poor  Monitoring  and  Supervision 
Subscale 

✓ 

✓ 

V 

APQ — Positive  Parenting  Subscale 

✓ 

✓ 

Child-Centered  Social  Control 

✓ 

Confusion,  Hubbub,  and  Order 

✓ 

V 

✓ 

Community  Resources 

✓ 

Community  Risk 

✓ 

V 

Parent  and  Teacher  Involvement 

✓ 

V 

Child  scales 

Altruistic  Behavior 

✓ 

✓ 

V 

Positive  Social  Behavior 

✓ 

✓ 

V 

✓ 

✓ 

✓ 

Problem  Behavior 

✓ 

✓ 

Teacher-reported 

Child  scales 

Academic  Competence  and  Motivation 

✓ 

✓ 

ADHD-Related  Behavior 

✓ 

✓ 

✓ 

✓ 

✓ 

Altruistic  Behavior 

✓ 

V 

✓ 

Positive  Social  Behavior 

V 

✓ 

Problem  Behavior 

✓ 

✓ 

✓ 

Parent  and  Teacher  Involvement 

✓ 

✓ 

✓ 

NOTE:  Abbreviations  are 

ABC:  Academic  and  Behavioral  Competencies  Program 
CSP:  Competence  Support  Program 
LBW:  Love  In  a Big  World 
PA:  Positive  Action 

PATHS:  Promoting  Alternative  Thinking  Strategies 

4Rs:  The  4Rs  Program  (Reading,  Writing,  Respect,  and  Resolution) 

SS:  Second  Step 

ADHD:  Attention  deficit  hyperactivity  disorder 
APQ:  Alabama  Parenting  Questionnaire 
•S : Covariate  used 
Blank  cell:  Covariate  not  used 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 


B-29 


Appendix  B:  Technical  Notes 


Table  B.12.  Covariates  selected  for  teacher-reported  school  outcomes,  for  the  combined- 
program  and  program-specific  samples 


Combined- 

program 


Potential  covariate 

sample 

ABC 

PA 

SS 

4Rs 

CSP 

PATHS 

LBW 

Total  number 

8 

8 

7 

8 

6 

9 

5 

5 

Teacher-reported 

Female 

V 

✓ 

✓ 

✓ 

✓ 

✓ 

✓ 

V 

Black  (non-Hispanic) 

V 

✓ 

✓ 

✓ 

✓ 

✓ 

✓ 

V 

Hispanic 

V 

✓ 

✓ 

✓ 

✓ 

✓ 

✓ 

V 

Other  ethnicity 

V 

✓ 

✓ 

✓ 

✓ 

✓ 

✓ 

V 

Total  teaching  experience 

V 

✓ 

✓ 

✓ 

✓ 

✓ 

✓ 

Total  experience  in  current  school 

✓ 

V 

Regular  certificate 

s 

✓ 

✓ 

✓ 

✓ 

Other  certificate 

s 

✓ 

✓ 

✓ 

✓ 

Highest  degree — Bachelor’s 

V 

✓ 

✓ 

✓ 

✓ 

NOTE:  Abbreviations  are 

ABC:  Academic  and  Behavioral  Competencies  Program 
CSP:  Competence  Support  Program 
LBW:  Love  In  a Big  World 
PA:  Positive  Action 

PATHS:  Promoting  Alternative  Thinking  Strategies 

4Rs:  The  4Rs  Program  (Reading,  Writing,  Respect,  and  Resolution) 

SS:  Second  Step 

v' : Covariate  used 

Blank  cell:  Covariate  not  used 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Table  B.13.  Adjusted  R2  values  from  models  using  spring  2005  (Year  1)  outcomes,  with  pretest 
only  and  with  the  full  covariate  set,  for  the  combined-program  sample 


Child  level Collapsed  to  school  level 

Full  Full 

covariate  covariate 


Outcome 

Pretest  only 

set 

Pretest  only 

set 

Child-reported 

Empathy 

0.19 

0.24 

0.34 

0.56 

Student  Afraid  at  School 

0.13 

0.18 

0.64 

0.76 

Altruistic  Behavior 

0.20 

0.23 

0.60 

0.67 

Engagement  with  Learning 

0.09 

0.13 

0.33 

0.49 

Negative  School  Orientation 

0.20 

0.26 

0.83 

0.88 

Problem  Behavior 

0.26 

0.31 

0.71 

0.82 

Self-Efficacy  for  Peer  Interactions 

0.20 

0.21 

0.22 

0.54 

Victimization  at  School 

0.20 

0.21 

0.47 

0.60 

Normative  Beliefs  About  Aggression 

0.11 

0.15 

0.39 

0.67 

Positive  School  Orientation 

0.18 

0.24 

0.71 

0.83 

Primary  caregiver-reported 

Altruistic  Behavior 

0.22 

0.23 

0.53 

0.65 

Positive  Social  Behavior 

0.43 

0.46 

0.74 

0.83 

Problem  Behavior 

0.37 

0.40 

0.51 

0.49 

Teacher-reported  (child) 

ADHD-Related  Behavior 

0.52 

0.54 

0.64 

0.71 

Academic  Competence  and  Motivation 

0.60 

0.62 

0.85 

0.88 

Altruistic  Behavior 

0.27 

0.28 

0.32 

0.28 

Positive  Social  Behavior 

0.55 

0.58 

0.81 

0.87 

Problem  Behavior 

0.51 

0.54 

0.78 

0.84 

Teacher-reported  (school) 

Student  Support  for  Teachers 

0.62 

0.63 

0.91 

0.91 

Feelings  of  Safety 

0.47 

0.48 

0.78 

0.80 

NOTE:  ADHD  is  the  abbreviation  for  attention  deficit  hyperactivity  disorder. 
SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Table  B.14.  Adjusted  R2  values  from  models  using  spring  2005  (Year  1)  outcomes  with  pretest  only  and  with  the  full  covariate  set,  by 
site 


LBW 

CSP 

SS 

ABC 

PATHS 

PA 

4Rs 

Pre- 

Pre- 

Pre- 

Pre- 

Pre- 

Pre- 

Pre- 

test 

Full 

test 

Full 

test 

Full 

test 

Full 

test 

Full 

test 

Full 

test 

Full 

Outcome 

only 

set 

only 

set 

only 

set 

only 

set 

only 

set 

only 

set 

only 

set 

Child-reported 

Empathy 

0.21 

0.24 

0.19 

0.27 

0.25 

0.26 

0.22 

0.24 

0.19 

0.26 

0.11 

0.18 

0.13 

0.17 

Student  Afraid  at  School 

0.05 

0.08 

0.10 

0.15 

0.20 

0.21 

0.09 

0.13 

0.15 

0.20 

0.07 

0.11 

0.09 

0.13 

Altruistic  Behavior 

0.17 

0.18 

0.20 

0.22 

0.24 

0.28 

0.30 

0.32 

0.22 

0.25 

0.10 

0.13 

0.13 

0.14 

Engagement  with  Learning 

0.13 

0.16 

0.03 

0.04 

0.11 

0.15 

0.09 

0.15 

0.11 

0.14 

0.06 

0.10 

0.12 

0.14 

Negative  School  Orientation 

0.19 

0.20 

0.14 

0.16 

0.18 

0.21 

0.15 

0.17 

0.20 

0.32 

0.08 

0.15 

0.14 

0.20 

Problem  Behavior 

0.23 

0.26 

0.18 

0.26 

0.28 

0.30 

0.29 

0.33 

0.28 

0.35 

0.10 

0.17 

0.27 

0.32 

Self-Efficacy  for  Peer  Interactions 

0.19 

0.20 

0.14 

0.14 

0.25 

0.27 

0.23 

0.23 

0.25 

0.26 

0.16 

0.21 

0.17 

0.17 

Victimization  at  School 

0.18 

0.20 

0.17 

0.18 

0.21 

0.22 

0.19 

0.21 

0.27 

0.27 

0.16 

0.18 

0.13 

0.13 

Normative  Beliefs  About  Aggression 

0.06 

0.06 

0.07 

0.15 

0.02 

0.03 

0.14 

0.19 

0.12 

0.17 

0.05 

0.11 

0.19 

0.20 

Positive  School  Orientation 

0.13 

0.12 

0.11 

0.17 

0.20 

0.20 

0.23 

0.27 

0.13 

0.26 

0.07 

0.16 

0.20 

0.27 

Primary  caregiver-reported 

Altruistic  Behavior 

0.14 

0.20 

0.18 

0.22 

0.23 

0.34 

0.19 

0.24 

0.26 

0.32 

0.19 

0.22 

0.20 

0.25 

Positive  Social  Behavior 

0.35 

0.39 

0.42 

0.46 

0.47 

0.50 

0.54 

0.56 

0.50 

0.54 

0.40 

0.46 

0.32 

0.36 

Problem  Behavior 

0.33 

0.40 

0.38 

0.43 

0.50 

0.53 

0.37 

0.40 

0.43 

0.51 

0.40 

0.44 

0.22 

0.26 

Teacher-reported  (child) 

ADHD-Related  Behavior 
Academic  Competence  and 

0.48 

0.54 

0.52 

0.53 

0.55 

0.57 

0.51 

0.54 

0.57 

0.59 

0.46 

0.54 

0.50 

0.54 

Motivation 

0.53 

0.56 

0.61 

0.63 

0.70 

0.72 

0.51 

0.55 

0.69 

0.72 

0.51 

0.56 

0.54 

0.57 

Altruistic  Behavior 

0.17 

0.19 

0.34 

0.36 

0.28 

0.34 

0.28 

0.33 

0.36 

0.38 

0.15 

0.20 

0.31 

0.35 

Positive  Social  Behavior 

0.51 

0.56 

0.42 

0.47 

0.55 

0.59 

0.51 

0.56 

0.58 

0.64 

0.54 

0.60 

0.52 

0.54 

Problem  Behavior 

0.38 

0.42 

0.47 

0.50 

0.47 

0.51 

0.45 

0.52 

0.67 

0.70 

0.50 

0.56 

0.50 

0.52 

Teacher-reported  (school) 

Student  Support  for  Teachers 

0.42 

0.43 

0.43 

0.44 

0.31 

0.36 

0.64 

0.63 

0.70 

0.70 

0.61 

0.62 

0.50 

0.52 

Feelings  of  Safety 

0.26 

0.29 

0.22 

0.30 

0.32 

0.36 

0.52 

0.52 

0.48 

0.49 

0.38 

0.36 

0.53 

0.53 

NOTE:  Abbreviations  are 

ABC:  Academic  and  Behavioral  Competencies  Program  PA:  Positive  Action 

CSP:  Competence  Support  Program  PATHS:  Promoting  Alternative  Thinking  Strategies 

LBW:  Love  In  a Big  World  4Rs:  The  4Rs  Program  (Reading,  Writing,  Respect,  and  Resolution) 

SS:  Second  Step  ADHD:  Attention  deficit  hyperactivity  disorder 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Table  B.15.  Adjusted  R2  values  from  models  using  spring  2006  (Year  2)  outcomes,  with  pretest 
only  and  with  the  full  covariate  set,  for  the  combined-program  sample 

Child  level Collapsed  to  school  level 


Full  Full 

covariate  covariate 


Outcome 

Pretest  only 

set 

Pretest  only 

set 

Child-reported 

Empathy 

0.11 

0.16 

0.32 

0.48 

Student  Afraid  at  School 

0.07 

0.12 

0.51 

0.65 

Altruistic  Behavior 

0.09 

0.11 

0.34 

0.42 

Engagement  with  Learning 

0.03 

0.07 

0.25 

0.30 

Negative  School  Orientation 

0.13 

0.19 

0.71 

0.72 

Problem  Behavior 

0.16 

0.24 

0.62 

0.77 

Self-Efficacy  for  Peer  Interactions 

0.10 

0.13 

0.36 

0.58 

Victimization  at  School 

0.08 

0.10 

0.16 

0.18 

Normative  Beliefs  About  Aggression 

0.08 

0.14 

0.30 

0.57 

Positive  School  Orientation 

0.13 

0.18 

0.63 

0.69 

Primary  caregiver-reported 

Altruistic  Behavior 

0.19 

0.21 

0.43 

0.67 

Positive  Social  Behavior 

0.32 

0.37 

0.42 

0.71 

Problem  Behavior 

0.27 

0.30 

0.37 

0.52 

Teacher-reported  (child) 

ADHD-Related  Behavior 

0.23 

0.29 

0.31 

0.46 

Academic  Competence  and  Motivation 

0.37 

0.42 

0.62 

0.78 

Altruistic  Behavior 

0.06 

0.09 

0.09 

0.01 

Positive  Social  Behavior 

0.22 

0.32 

0.35 

0.64 

Problem  Behavior 

0.24 

0.31 

0.62 

0.76 

Teacher-reported  (school) 

Student  Support  for  Teachers 

0.49 

0.49 

0.85 

0.85 

Feelings  of  Safety 

0.37 

0.38 

0.68 

0.69 

NOTE:  ADHD  is  the  abbreviation  for  attention  deficit  hyperactivity  disorder. 
SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Table  B.16.  Adjusted  R2  values  from  models  using  spring  2006  (Year  2)  outcomes  with  pretest  only  and  with  the  full  covariate  set 
selected  for  each  program,  by  program 


ABC 

PA 

SS 

4Rs 

CSP 

PATHS 

LBW 

Pre- 

Pre- 

Pre- 

Pre- 

Pre- 

Pre- 

Pre- 

test 

Full 

test 

Full 

test 

Full 

test 

Full 

test 

Full 

test 

Full 

test 

Full 

Outcome 

only 

set 

only 

set 

only 

set 

only 

set 

only 

set 

only 

set 

only 

set 

Child-reported 

Empathy 

0.12 

0.17 

0.04 

0.12 

0.09 

0.17 

0.07 

0.12 

0.06 

0.12 

0.12 

0.22 

0.08 

0.10 

Student  Afraid  at  School 

0.07 

0.12 

0.01 

0.04 

0.03 

0.08 

0.03 

0.04 

0.02 

0.06 

0.08 

0.18 

0.01 

0.04 

Altruistic  Behavior 

0.10 

0.13 

0.06 

0.08 

0.06 

0.11 

0.05 

0.06 

0.09 

0.14 

0.11 

0.12 

0.05 

0.07 

Engagement  with  Learning 

0.03 

0.04 

0.02 

0.04 

0.04 

0.05 

0.03 

0.09 

0.01 

0.05 

0.05 

0.10 

0.03 

0.05 

Negative  School  Orientation 

0.04 

0.07 

0.04 

0.10 

0.08 

0.12 

0.03 

0.09 

0.03 

0.06 

0.20 

0.29 

0.08 

0.09 

Problem  Behavior 

0.13 

0.20 

0.04 

0.16 

0.10 

0.16 

0.15 

0.23 

0.06 

0.15 

0.15 

0.28 

0.06 

0.11 

Self-Efficacy  for  Peer  Interactions 

0.10 

0.15 

0.05 

0.17 

0.08 

0.10 

0.09 

0.09 

0.09 

0.10 

0.17 

0.21 

0.10 

0.11 

Victimization  at  School 

0.10 

0.11 

0.04 

0.07 

0.12 

0.11 

0.07 

0.07 

0.06 

0.11 

0.11 

0.11 

0.04 

0.06 

Normative  Beliefs  About  Aggression 

0.07 

0.12 

0.03 

0.09 

0.01 

0.06 

0.08 

0.10 

0.02 

0.09 

0.04 

0.16 

0.14 

0.15 

Positive  School  Orientation 

0.12 

0.17 

0.06 

0.24 

0.03 

0.03 

0.07 

0.12 

0.03 

0.10 

0.17 

0.33 

0.06 

0.06 

Primary  caregiver-reported 

Altruistic  Behavior 

0.15 

0.16 

0.18 

0.19 

0.22 

0.26 

0.17 

0.17 

0.13 

0.12 

0.24 

0.32 

0.10 

0.15 

Positive  Social  Behavior 

0.36 

0.41 

0.35 

0.39 

0.33 

0.42 

0.16 

0.19 

0.26 

0.33 

0.36 

0.39 

0.33 

0.42 

Problem  Behavior 

0.34 

0.36 

0.31 

0.32 

0.27 

0.33 

0.09 

0.12 

0.23 

0.30 

0.41 

0.45 

0.26 

0.32 

Teacher-reported  (child) 

ADHD-Related  Behavior 

0.22 

0.25 

0.14 

0.24 

0.27 

0.33 

0.20 

0.26 

0.25 

0.29 

0.33 

0.36 

0.20 

0.28 

Academic  Competence  and  Motivation 

0.35 

0.42 

0.27 

0.34 

0.39 

0.47 

0.27 

0.31 

0.30 

0.42 

0.54 

0.57 

0.30 

0.35 

Altruistic  Behavior 

0.02 

0.05 

0.00 

0.08 

0.00 

0.04 

0.00 

0.03 

0.02 

0.05 

0.01 

0.00 

0.02 

0.10 

Positive  Social  Behavior 

0.17 

0.27 

0.19 

0.30 

0.16 

0.29 

0.26 

0.32 

0.14 

0.26 

0.33 

0.46 

0.12 

0.21 

Problem  Behavior 

0.20 

0.28 

0.20 

0.30 

0.18 

0.32 

0.20 

0.25 

0.25 

0.29 

0.36 

0.46 

0.17 

0.21 

Teacher-reported  (school) 

Student  Support  for  Teachers 

0.28 

0.26 

0.53 

0.54 

0.30 

0.28 

0.31 

0.29 

0.29 

0.31 

0.56 

0.56 

0.18 

0.23 

Feelings  of  Safety 

0.42 

0.42 

0.48 

0.50 

0.12 

0.11 

0.36 

0.35 

0.12 

0.18 

0.41 

0.41 

0.16 

0.21 

NOTE:  Abbreviations  are 

ABC:  Academic  and  Behavioral  Competencies  Program 
CSP:  Competence  Support  Program 
LBW:  Love  In  a Big  World 
SS:  Second  Step 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research 


PA:  Positive  Action 

PATHS:  Promoting  Alternative  Thinking  Strategies 
4Rs:  The  4Rs  Program  (Reading,  Writing,  Respect,  and  Resolution) 
ADHD:  Attention  deficit  hyperactivity  disorder 
Program. 
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Table  B.17.  Adjusted  R2  values  from  models  using  spring  2007  (Year  3)  outcomes,  with  pretest 
only  and  with  the  full  covariate  set,  for  the  combined-program  sample 


Outcome 

Child  level 

Full 
covariate 
Pretest  only  set 

Collapsed  to  school  level 
Full 
covariate 
Pretest  only  set 

Child-reported 

Empathy 

0.09 

0.14 

0.40 

0.67 

Student  Afraid  at  School 

0.06 

0.12 

0.51 

0.70 

Altruistic  Behavior 

0.06 

0.08 

0.41 

0.56 

Engagement  with  Learning 

0.02 

0.07 

0.01 

0.29 

Negative  School  Orientation 

0.11 

0.17 

0.53 

0.64 

Problem  Behavior 

0.14 

0.23 

0.55 

0.74 

Self-Efficacy  for  Peer  Interactions 

0.06 

0.08 

0.15 

0.30 

Victimization  at  School 

0.06 

0.07 

0.25 

0.28 

Normative  Beliefs  About  Aggression 

0.06 

0.11 

0.35 

0.46 

Positive  School  Orientation 

0.11 

0.16 

0.47 

0.67 

Primary  caregiver-reported 

Altruistic  Behavior 

0.16 

0.20 

0.40 

0.74 

Positive  Social  Behavior 

0.26 

0.32 

0.54 

0.69 

Problem  Behavior 

0.19 

0.23 

0.31 

0.64 

Teacher-reported  (child) 

ADHD-Related  Behavior 

0.20 

0.26 

0.05 

0.69 

Academic  Competence  and  Motivation 

0.32 

0.37 

0.49 

0.70 

Altruistic  Behavior 

0.06 

0.07 

0.10 

0.20 

Positive  Social  Behavior 

0.22 

0.30 

0.30 

0.60 

Problem  Behavior 

0.22 

0.28 

0.42 

0.61 

Teacher-reported  (school) 

Student  Support  for  Teachers 

0.47 

0.47 

0.78 

0.77 

Feelings  of  Safety 

0.30 

0.30 

0.50 

0.48 

NOTE:  ADHD  is  the  abbreviation  for  attention  deficit  hyperactivity  disorder. 
SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 


B-35 


B-36 


Table  B.18.  Adjusted  R2  values  from  models  using  spring  2007  (Year  3)  outcomes,  with  pretest  only  and  with  the  full  covariate  set 
selected  for  each  program,  by  program 


LBW 

CSP 

SS 

ABC 

PATHS 

PA 

4Rs 

Pre- 

Pre- 

Pre- 

Pre- 

Pre- 

Pre- 

Pre- 

test 

Full 

test 

Full 

test 

Full 

test 

Full 

test 

Full 

test 

Full 

test 

Full 

Outcome 

only 

set 

only 

set 

only 

set 

only 

set 

only 

set 

only 

set 

only 

set 

Child-reported 

Empathy 

0.08 

0.13 

0.07 

0.14 

0.06 

0.15 

0.05 

0.06 

0.03 

0.11 

0.04 

0.11 

0.03 

0.09 

Student  Afraid  at  School 

0.03 

0.06 

0.01 

0.05 

0.03 

0.12 

0.04 

0.10 

0.06 

0.17 

0.01 

0.07 

0.02 

0.05 

Altruistic  Behavior 

0.04 

0.08 

0.08 

0.08 

0.05 

0.09 

0.05 

0.08 

0.03 

0.09 

0.04 

0.05 

0.03 

0.04 

Engagement  with  Learning 

0.06 

0.07 

0.00 

0.04 

0.09 

0.13 

0.02 

0.04 

0.05 

0.12 

0.02 

0.07 

0.01 

0.07 

Negative  School  Orientation 

0.05 

0.08 

0.04 

0.10 

0.07 

0.14 

0.07 

0.08 

0.19 

0.28 

0.04 

0.12 

0.01 

0.05 

Problem  Behavior 

0.12 

0.17 

0.05 

0.16 

0.07 

0.15 

0.11 

0.13 

0.22 

0.33 

0.02 

0.18 

0.07 

0.17 

Self-Efficacy  for  Peer  Interactions 

0.04 

0.05 

0.08 

0.11 

0.05 

0.09 

0.07 

0.11 

0.09 

0.12 

0.04 

0.09 

0.05 

0.06 

Victimization  at  School 

0.02 

0.02 

0.05 

0.07 

0.07 

0.09 

0.04 

0.07 

0.13 

0.17 

0.02 

0.07 

0.02 

0.03 

Normative  Beliefs  About  Aggression 

0.07 

0.12 

0.01 

0.14 

0.01 

0.06 

0.01 

0.05 

0.03 

0.13 

0.03 

0.08 

0.07 

0.12 

Positive  School  Orientation 

0.07 

0.07 

0.02 

0.08 

0.02 

0.07 

0.09 

0.12 

0.09 

0.28 

0.04 

0.18 

0.02 

0.07 

Primary  caregiver-reported 

Altruistic  Behavior 

0.08 

0.21 

0.15 

0.16 

0.13 

0.24 

0.13 

0.19 

0.20 

0.28 

0.09 

0.16 

0.17 

0.17 

Positive  Social  Behavior 

0.22 

0.29 

0.29 

0.33 

0.31 

0.35 

0.23 

0.28 

0.35 

0.40 

0.22 

0.28 

0.12 

0.19 

Problem  Behavior 

0.23 

0.26 

0.19 

0.24 

0.21 

0.28 

0.17 

0.23 

0.25 

0.28 

0.18 

0.20 

0.09 

0.17 

Teacher-reported  (child) 

ADHD-Related  Behavior 

0.19 

0.25 

0.25 

0.32 

0.26 

0.28 

0.14 

0.20 

0.24 

0.28 

0.13 

0.25 

0.15 

0.26 

Academic  Competence  and  Motivation 

0.28 

0.33 

0.25 

0.36 

0.35 

0.41 

0.27 

0.32 

0.34 

0.40 

0.20 

0.27 

0.27 

0.33 

Altruistic  Behavior 

0.00 

0.02 

0.06 

0.11 

0.01 

0.05 

0.00 

0.01 

0.01 

0.05 

0.03 

0.07 

0.00 

0.06 

Positive  Social  Behavior 

0.14 

0.22 

0.21 

0.32 

0.15 

0.27 

0.11 

0.15 

0.27 

0.41 

0.17 

0.29 

0.19 

0.24 

Problem  Behavior 

0.19 

0.24 

0.19 

0.26 

0.09 

0.21 

0.15 

0.19 

0.36 

0.48 

0.24 

0.31 

0.13 

0.18 

Teacher-reported  (school) 

Student  Support  for  Teachers 

0.11 

0.14 

0.19 

0.22 

0.27 

0.27 

0.37 

0.43 

0.55 

0.54 

0.58 

0.59 

0.23 

0.22 

Feelings  of  Safety 

0.21 

0.20 

0.17 

0.27 

0.08 

0.11 

0.31 

0.30 

0.32 

0.29 

0.39 

0.41 

0.19 

0.20 

NOTE:  Abbreviations  are 

ABC:  Academic  and  Behavioral  Competencies  Program  PA:  Positive  Action 

CSP:  Competence  Support  Program  PATHS:  Promoting  Alternative  Thinking  Strategies 

LBW:  Love  In  a Big  World  4Rs:  The  4Rs  Program  (Reading,  Writing,  Respect,  and  Resolution) 

SS:  Second  Step  ADHD:  Attention  deficit  hyperactivity  disorder 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Construction  of  Sample  Weights 

Two  sets  of  weights  were  constructed  for  the  year-by-year  SACD  analysis:  (1)  base  weights  assigned  equal 
weight  to  each  program  and  to  each  school  within  a program,  and  (2)  adjusted  weights  modified  the  base 
weights  for  study  nonconsent,  interview  nonresponse,  and  students  who  entered  the  study  schools  after  fall 
2004  (new  entrants).  The  weighting  approach  means  that  the  combined-program  impacts  were  the  simple 
averages  of  the  program  impacts  (based  on  the  same  covariates  within  the  regression),  the  program  impacts 
were  the  simple  averages  of  the  school  impacts,  and  the  school  impacts  were  weighted  averages  of  the 
impacts  in  each  classroom,  with  weights  proportional  to  classroom  size.  The  weighting  approach  was  chosen 
to  generate  the  average  impact  across  schools  (and  programs)  in  the  sample.  This  approach  was  grounded  in 
the  study  design,  in  which  random  assignment  was  conducted  at  the  school  level  within  programs.  This 
weighting  approach  produced  unbiased  estimates  of  the  average  treatment  effect  for  the  study  schools  and 
programs  that  were  purposively  selected  for  the  evaluation.  This  approach  was  consistent  with  an  analysis  in 
which  the  84  school  means  were  used  in  the  regression  models  (that  is,  a between-school  analysis),  and  the 
model  included  treatment-by-program  interactions  that  were  weighted  equally  to  estimate  the  overall  cross- 
site impact. 

Base  Weights 

For  each  spring  (2005-07)  follow-up  instrument,  base  weights  were  constructed  with  the  following  properties: 

• Within  each  school,  each  survey  respondent  had  the  same  weight. 

• Within  each  program,  the  weights  in  each  school  summed  to  the  same  value. 

• The  weights  in  each  program  summed  to  the  same  value  (equal  to  the  average  program  sample  size). 

• The  sum  of  all  the  weights  was  equal  to  twice  the  original  sample  size  (because,  through  random 
assignment,  the  control  and  treatment  groups  each  represented  the  sample). 

To  satisfy  these  objectives,  the  base  weights  for  the  student-level  outcome  measures  were  calculated  using  the 
following  formula: 

base«eightsr  = 

where  basemightsp  is  the  weight  for  a child  respondent  in  school  ,f  and  program  p,  nsp  is  the  average  number 

of  child  respondents  per  study  school,  % is  the  number  of  respondents  in  school  s and  program  p,  schp  is 
the  average  number  of  schools  per  program,  and  schp'is  the  number  of  schools  in  program  p. 

Equation  (1)  was  also  used  to  construct  base  weights  for  the  school-level  outcome  measures  from  the 
Teacher  Report  on  Classroom  and  School  data.  However  in  this  case,  the  first  term  on  the  right-hand  side  of 
equation  (1)  refers  to  the  number  of  third-,  fourth-,  and  fifth-grade  teacher  respondents  in  the  school  instead 
of  the  number  of  child  respondents.  That  is,  the  numerator  of  the  first  term  is  the  average  number  of  teacher 
respondents  per  study  school,  and  the  denominator  is  the  number  of  teacher  respondents  in  school  s and 
program  p. 

Base  weights  were  constructed  using  the  total  number  of  respondents  who  completed  each  instrument, 
regardless  of  the  incidence  of  missing  data  for  specific  variables  in  the  instruments. 

Adjusted  Weights 

To  account  for  study  nonconsent  and  interview  nonresponse,  adjusted  weights  were  constructed  for  the 
student-level  and  the  school-level  outcome  measures  so  that  the  impact  estimates  could  be  generalized  to  the 
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full  sample  universe,  rather  than  to  those  with  follow-up  data  only.  For  the  student-level  outcome  measures, 
the  original  cohort  children  (stayers)  in  a classroom  who  responded  to  an  instrument  were  assumed  to  be 
representative  of  all  original  cohort  children  (stayers)  in  that  classroom.  This  approach  adjusted  for  different 
survey  response  rates  across  classrooms  and  used  the  following  equation: 

(2)  ad]  wt  childcsp 

p \ncspJ  VspJ  \schpJ 

Thus,  adj_wt_childcsp  is  the  adjusted  weight  based  on  enrollment  for  an  original  cohort  child  (stayer)  in  school  r 
and  program  p,  nCSp  refers  to  the  average  number  of  fifth-grade  stayers  per  classroom  c in  school  s at 
program  p,  ncsp  refers  to  the  number  of  stayer  respondents  in  a specific  classroom  c in  school  s at  program  p, 

CSp  refers  to  the  average  number  of  classrooms  in  schools  across  programs,  and  Csp  refers  to  the  number  of 
classrooms  in  school  s at  program  p. 

Because  new  entrants  were  less  likely  to  get  parental  consent  than  the  original  cohort  of  children,  the  adjusted 
weights  for  the  child  assessments  were  constructed  separately  for  the  two  groups  of  children.  There  were  very 
few  new  entrants  within  some  classrooms,  so  the  nonresponse  and  nonconsent  adjustments  were  conducted 
at  the  school  level  for  this  group.  The  equation  for  the  adjusted  weights  for  new  entrants,  which  does  not 
include  a classroom-level  component,  is  as  follows: 


(3)  adj  _wt_new_entrantsp  = 


where  adj_wt_new_entrantsp  is  the  adjusted  weight  based  on  enrollment  for  a new  entrant  child  in  school  s and 
program  p,  nsp  is  the  average  population  of  new  entrants  per  study  school,  and  % is  the  number  of  new 
entrant  respondents  in  school  s and  program  p. 

For  the  school-level  outcome  measures,  the  responding  teachers  were  assumed  to  be  representative  of  third-, 
fourth-,  and  fifth-grade  teachers  in  that  school,  which  means  that  there  is  no  classroom-level  component  in 
the  construction  of  the  Teacher  Report  on  Classroom  and  School  adjusted  weights.  The  equation  for  these 
weights  is  as  follows: 


(4)  adj_wt_TRCSsp  = 


where  nsp  is  the  average  population  of  teachers  per  study  school  and  ti#  is  the  number  of  teacher 
respondents  in  school  s and  program  p. 

Weights  for  the  Growth  Curve  Analysis 

The  growth  curve  analysis  included  data  from  every  survey  wave.  Weights  were  constructed  specifically  for 
the  growth  curve  analysis  in  order  to  maintain  the  equal  representation  of  programs  in  the  analysis.  The 
growth  curve  weights  were  constructed  only  at  the  child  level  because  the  growth  curve  analysis  was  only 
performed  for  child-level  outcomes.  Growth  curve  weights  were  constructed  with  the  following  properties: 

• Within  each  school  and  survey  period,  each  student  had  the  same  weight. 

• Within  each  program  and  survey  period,  the  weights  in  each  school  summed  to  the  same  value. 

• Within  each  survey  period,  the  weights  in  each  program  summed  to  the  same  value. 

• The  weights  within  each  survey  period  summed  to  the  same  value  for  all  survey  periods. 
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The  formula  for  the  growth  weights  is  as  follows: 


(5) 


GCweighttip  = 


(r\  ( nt,kids\  ( scht 

W V nt,i,p  J \scht,P 


where  GC weight n is  the  weight  for  students  at  survey  period  /,  in  school  / in  program  p;  lit, kids  represents 
the  average  number  of  students  in  a school  at  survey  period  t;  ntAS  represents  the  number  of  students  in 

school  i in  program  r at  survey  period  t;  scht  represents  the  average  number  of  schools  in  a program  at 
survey  period  t,  and  scht,s  represents  the  number  of  schools  in  program  r at  survey  period  /.  The  average 

number  of  students  in  a survey  period  is  represented  by  r , while  rt  represents  the  number  of  students  within 
survey  period  /. 


Sensitivity  Analysis 

The  hierarchical  linear  modeling  (HLM)  framework  for  estimating  the  impacts  of  the  SACD  programs  on  key 
child  and  school  outcomes  estimated  impacts  using  regression  models,  where  the  standard  errors  of  the 
impact  estimates  accounted  for  design  effects  due  to  school-level  clustering  and  precision  gains  due  to  the 
inclusion  of  baseline  covariates  in  the  models  (Bryk  and  Raudenbush  1992). 

To  implement  the  HLM  framework,  it  was  necessary  to  make  decisions  about  key  model  parameter 
specifications  and  estimation  methods.  A set  of  sensitivity  analyses  was  done  to  determine  if  these  model 
assumptions  affected  the  results  from  the  combined -program  impact  analysis  for  the  18  student-level 
outcome  measures.  Ten  sensitivity  tests  were  done  and  these  included  the  following  variations: 

• No  initial  covariates  were  included  in  the  regression  models. 

• The  sample  weights  were  not  adjusted  for  study  nonconsent  and  interview  nonresponse  and  when 
models  were  estimated  without  weights. 

• The  error  structure  included  classroom-level  random  effects. 

• The  error  structure  accounted  for  the  pairwise  matching  of  schools. 

• The  pretests  were  treated  as  dependent  variables  rather  than  as  covariates. 

• The  impacts  were  estimated  using  alternative  estimation  routines  (statistical  software  packages). 

• Missing  outcome  measures  were  imputed  using  multiple  imputation  procedures. 

• Combined-program  impact  estimates  were  obtained  by  averaging  the  program-level  impact  estimates. 

• Various  sets  of  covariates  were  included  in  the  regression  models. 

• New  entrants  were  excluded  from  the  analysis. 

The  number  of  sensitivity  analyses  done  declined  each  year  as  the  pattern  of  impact  results  from  the  original 
model  proved  robust  to  a variety  of  model  specifications.  All  10  sensitivity  analyses  were  done  with  the  Year 
1 data.  The  first  9 sensitivity  tests  were  done  with  the  Year  2 data.  The  sample  of  new  entrants  grew  large 
enough  in  Years  2 and  3 to  do  separate  analyses  of  new  entrants  versus  stayers  (in  place  of  the  10th  sensitivity 
test)  and  these  were  included  in  the  subgroup  analyses  discussed  in  chapter  1.  The  1st,  2nd  (except  for  the 
estimation  without  weights),  3rd,  and  4th  sensitivity  tests  were  done  using  the  Year  3 data.  The  sensitivity 
analyses  and  their  results  are  discussed  below. 
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Excluding  Covariates 

Under  the  experimental  design,  the  inclusion  of  baseline  covariates  was  not  required  to  obtain  unbiased 
impact  estimates.  However,  covariates  were  used  because  they  increased  the  precision  of  the  impact  estimates 
by  explaining  some  of  the  variations  in  the  outcome  measures  between  schools  and  across  students  within 
schools.  In  addition,  covariates  adjusted  for  residual  differences  between  the  baseline  characteristics  of 
treatment  and  control  group  members  due  to  random  sampling,  study  nonconsent,  and  interview 
nonresponse. 

As  a specification  test,  the  combined-program  models  were  estimated  with  no  covariates  except  for  site 
binary  variables  (table  B.19).  As  with  the  original  impact  estimates,  no  significant  impacts  were  found  from 
these  simple  differences-in-means  estimates. 
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Table  B.19.  Combined-program  impacts  in  effect  size  units  when  baseline  covariates  are 
excluded  from  the  models 


Year  1 Year  2 Year  3 

(Spring  2005)  (Spring  2006)  (Spring  2007) 


p-value  of  p-value  of  p-value  of 


Scale-Report 

Impact1 

impact2 

Impact1 

impact2 

Impact1 

impact2 

Social  and  Emotional  Competence  Domain 

Self-Efficacy  for  Peer  Interactions-CR  (+) 

-0.05 

0.281 

-0.07 

0.164 

-0.04 

0.295 

Normative  Beliefs  About  Aggression-CR  (-) 

-0.01 

0.766 

-0.01 

0.912 

0.03 

0.673 

Empathy-CR  (+) 

0.06 

0.298 

-0.04 

0.524 

-0.07 

0.254 

Behavior  Domain 

Altruistic  Behavior-CR  (+) 

-0.06 

0.295 

-0.06 

0.208 

-0.02 

0.626 

Altruistic  Behavior-PCR  (+) 

0.05 

0.274 

0.01 

0.836 

-0.01 

0.861 

Altruistic  Behavior-TRS  (+) 

0.12 

0.253 

0.02 

0.910 

-0.05 

0.695 

Positive  Social  Behavior-PCR  (+) 

-0.01 

0.898 

0.04 

0.532 

0.01 

0.907 

Positive  Social  Behavior-TRS  (+) 

0.02 

0.830 

-0.05 

0.597 

-0.01 

0.917 

Problem  Behavior-CR  (-) 

0.02 

0.741 

0.03 

0.738 

0.08 

0.341 

Problem  Behavior-PCR  (+) 

-0.04 

0.483 

-0.07 

0.142 

-0.06 

0.283 

Problem  Behavior-TRS  (+) 

0.03 

0.762 

0.00 

0.989 

0.00 

0.994 

ADHD-Related  Behavior-TRS  (-) 

0.01 

0.871 

-0.01 

0.867 

-0.03 

0.610 

Academics  Domain 

Engagement  with  Learning-CR  (+) 

-0.06 

0.183 

-0.05 

0.217 

-0.06 

0.237 

Academic  Competence  and 

Motivation-TRS  (+) 

-0.06 

0.333 

-0.07 

0.298 

-0.07 

0.290 

Perceptions  of  School  Climate  Domain 

Positive  School  Orientation-CR  (+) 

0.01 

0.933 

0.00 

0.993 

-0.09 

0.363 

Negative  School  Orientation-CR  (-) 

-0.02 

0.735 

0.01 

0.942 

0.02 

0.753 

Student  Afraid  at  School-CR  (-) 

-0.04 

0.480 

-0.05 

0.488 

0.00 

0.981 

Victimization  at  School-CR  (-) 

-0.02 

0.637 

-0.04 

0.458 

0.00 

0.978 

1 1mpacts  are  in  effect  size  (standard  deviation)  units  and  were  calculated  by  dividing  the  estimated  impact  by  the  standard  deviation 
of  the  outcome  measure  for  the  control  group. 

2 The  p-value  is  from  a two-tailed  t test  to  gauge  the  statistical  significance  of  the  impact  estimate. 

NOTE:  Abbreviations  are 
CR:  Child  Report 
PCR:  Primary  Caregiver  Report 
TRS:  Teacher  Report  on  Student 
ADHD:  Attention  deficit  hyperactivity  disorder 

The  +/-  signs  in  parentheses  indicate  the  direction  of  a beneficial  outcome.  No  findings  were  found  statistically  significant  at  or  below 
the  .05  level.  All  impact  estimates  were  calculated  using  regression  models,  where  each  program  and  school  within  a program  was 
weighted  equally.  The  standard  errors  of  all  estimates  account  for  design  effects  due  to  unequal  weighting  and  the  clustering  of 
students  within  schools. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Employing  Alternative  Weighting  Schemes 

As  discussed  earlier  in  this  appendix,  the  impact  models  were  estimated  using  sample  weights.  The  weights 
were  constructed  so  that  each  site  and  each  school  within  a site  were  given  equal  weight  in  the  analysis,  and 
survey  respondents  were  assumed  to  represent  not  only  themselves  but  also  study  nonconsenters  and  survey 
nonrespondents  in  the  same  classroom  (by  new  entrant/original  cohort  status). 

To  examine  the  sensitivity  of  results  to  different  weighting  schemes,  models  were  estimated  using  weights  not 
adjusted  for  study  nonconsent  and  survey  nonresponse  for  each  of  the  3 years  (table  B.20).  In  addition, 
models  were  estimated  without  weights  for  Years  1 and  2 (table  B.21).  The  alternative  weighting  schemes  did 
not  change  the  statistical  significance  of  the  estimated  impacts  with  one  exception:  a detrimental  impact  was 
found  for  Altruistic  Behavior  (CR)  in  Year  1 when  weights  were  not  adjusted  for  study  nonconsent  and 
survey  nonresponse. 
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Table  B.20.  Combined-program  impacts  in  effect  size  units  when  weights  are  not  adjusted  for 
study  nonconsent  or  survey  nonresponse 


Year  1 Year  2 Year  3 

(Spring  2005)  (Spring  2006)  (Spring  2007) 


p-value  of  p-value  of  p-value  of 


Scale-Report 

Impact 

impact 

Impact 

impact 

Impact 

impact 

Social  and  Emotional  Competence  Domain 

Self-Efficacy  for  Peer  Interactions-CR  (+) 

-0.05 

0.236 

-0.07A 

0.073 

-0.04 

0.298 

Normative  Beliefs  About  Aggression-CR  (-) 

0.00 

0.950 

-0.02 

0.773 

0.02 

0.788 

Empathy-CR  (+) 

0.05 

0.272 

0.00 

0.962 

-0.03 

0.529 

Behavior  Domain 

Altruistic  Behavior-CR  (+) 

-0.07* 

0.041 

-0.04 

0.384 

-0.03 

0.452 

Altruistic  Behavior-PCR  (+) 

0.06A 

0.078 

-0.02 

0.530 

0.04 

0.445 

Altruistic  Behavior-TRS  (+) 

0.06 

0.483 

-0.01 

0.959 

-0.07 

0.548 

Positive  Social  Behavior-PCR  (+) 

0.00 

0.870 

0.05 

0.134 

0.04 

0.321 

Positive  Social  Behavior-TRS  (+) 

0.01 

0.767 

-0.02 

0.748 

0.04 

0.522 

Problem  Behavior-CR  (-) 

0.01 

0.819 

0.00 

0.942 

0.04 

0.396 

Problem  Behavior-PCR  (+) 

0.00 

0.985 

-0.06 

0.119 

-0.03 

0.504 

Problem  Behavior-TRS  (+) 

0.03 

0.503 

-0.04 

0.402 

-0.06 

0.279 

ADHD-Related  Behavior-TRS  (-) 

0.01 

0.866 

-0.05 

0.347 

-0.13 

0.122 

Academics  Domain 

Engagement  with  Learning-CR  (+) 

-0.05 

0.154 

-0.01 

0.721 

-0.04 

0.308 

Academic  Competence  and 

Motivation-TRS  (+) 

-0.02 

0.365 

0.00 

0.929 

0.01 

0.924 

Perceptions  of  School  Climate  Domain 

Positive  School  Orientation-CR  (+) 

0.03 

0.547 

0.04 

0.634 

-0.07 

0.387 

Negative  School  Orientation-CR  (-) 

-0.03 

0.320 

-0.04 

0.365 

-0.02 

0.726 

Student  Afraid  at  School-CR  (-) 

-0.05 

0.211 

-0.07 

0.145 

0.00 

0.970 

Victimization  at  School-CR  (-) 

0.00 

0.940 

-0.04 

0.391 

0.00 

0.959 

* Significantly  different  from  zero  at  the  .05  level. 

A Significantly  different  from  zero  at  the  .10  to  > .05  level. 

1 1mpacts  are  in  effect  size  (standard  deviation)  units  and  were  calculated  by  dividing  the  estimated  impact  by  the  standard  deviation 
of  the  outcome  measure  for  the  control  group. 

2 The  p-value  is  from  a two-tailed  t test  to  gauge  the  statistical  significance  of  the  impact  estimate. 

NOTE:  Abbreviations  are 
CR:  Child  Report 
PCR:  Primary  Caregiver  Report 
TRS:  Teacher  Report  on  Student 
ADHD:  Attention  deficit  hyperactivity  disorder 

The  +/-  signs  in  parentheses  indicate  the  direction  of  a beneficial  outcome.  All  impact  estimates  were  calculated  using  regression 
models,  where  each  program  and  school  within  a program  was  weighted  equally.  The  standard  errors  of  all  estimates  account  for 
design  effects  due  to  unequal  weighting  and  the  clustering  of  students  within  schools. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Table  B.21.  Combined-program  impacts  in  effect  size  units  when  weights  are  not  used 


Scale-Report 

Year  1 

(Spring  2005) 

p-value  of 
Impact1  impact2 

Year  2 

(Spring  2006) 

p-value  cf 
Impact1  impact2 

Social  and  Emotional  Competence  Domain 

Self-Efficacy  for  Peer  Interactions-CR  (+) 

-0.05 

0.211 

-0.07 

0.124 

Normative  Beliefs  About  Aggression-CR  (-) 

-0.01 

0.825 

0.00 

0.973 

Empathy-CR  (+) 

0.04 

0.474 

-0.04 

0.551 

Behavior  Domain 

Altruistic  Behavior-CR  (+) 

-0.07 

0.190 

-0.07 

0.181 

Altruistic  Behavior-PCR  (+) 

0.04 

0.402 

-0.01 

0.858 

Altruistic  Behavior-TRS  (+) 

0.11 

0.298 

0.01 

0.925 

Positive  Social  Behavior-PCR  (+) 

0.00 

0.974 

0.01 

0.851 

Positive  Social  Behavior-TRS  (+) 

0.01 

0.934 

-0.05 

0.571 

Problem  Behavior-CR  (-) 

0.02 

0.740 

0.02 

0.827 

Problem  Behavior-PCR  (+) 

-0.02 

0.672 

-0.07 

0.112 

Problem  Behavior-TRS  (+) 

0.04 

0.644 

-0.01 

0.946 

ADHD-Related  Behavior-TRS  (-) 

0.03 

0.636 

-0.02 

0.765 

Academics  Domain 

Engagement  with  Learning-CR  (+) 

-0.07 

0.120 

-0.04 

0.301 

Academic  Competence  and  Motivation-TRS  (+) 

-0.05 

0.410 

-0.06 

0.314 

Perceptions  of  School  Climate  Domain 

Positive  School  Orientation-CR  (+) 

0.01 

0.932 

0.01 

0.881 

Negative  School  Orientation-CR  (-) 

-0.01 

0.889 

-0.01 

0.870 

Student  Afraid  at  School-CR  (-) 

-0.04 

0.483 

-0.06 

0.336 

Victimization  at  School-CR  (-) 

-0.02 

0.725 

-0.05 

0.246 

1 1mpacts  are  in  effect  size  (standard  deviation)  units  and  were  calculated  by  dividing  the  estimated  impact  by  the  standard  deviation 
of  the  outcome  measure  for  the  control  group. 

2 The  p-value  is  from  a two-tailed  t test  to  gauge  the  statistical  significance  of  the  impact  estimate. 

NOTE:  Abbreviations  are 
CR:  Child  Report 
PCR:  Primary  Caregiver  Report 
TRS:  Teacher  Report  on  Student 
ADHD:  Attention  deficit  hyperactivity  disorder 

The  +/-  signs  in  parentheses  indicate  the  direction  of  a beneficial  outcome.  No  findings  were  found  statistically  significant  at  or  below 
the  .05  level.  All  impact  estimates  were  calculated  using  regression  models,  where  each  program  and  school  within  a program  was 
weighted  equally.  The  standard  errors  of  all  estimates  account  for  design  effects  due  to  unequal  weighting  and  the  clustering  of 
students  within  schools. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Including  Classroom-Level  Random  Effects  in  the  Error  Structure 

In  the  original  models,  adjustments  were  made  for  the  clustering  of  students  within  schools  but  not  for  the 
clustering  of  students  within  classrooms.  This  is  because  classrooms  were  not  sampled  for  the  study;  instead, 
all  classrooms  within  the  study  schools  were  included  in  the  evaluation.  Thus,  the  impact  estimates 
generalized  to  the  set  of  classrooms  and  teachers  that  were  in  the  study  schools  at  the  time  of  the  evaluation. 

An  alternative  view  would  be  that  the  estimated  impact  findings  generalized  to  a broader,  but  unknown, 
population  of  classrooms  within  the  study  schools.  To  accommodate  this  view,  classroom-level  random 
effects  were  included  in  the  models  using  a three-level  HLM  model  that  was  indexed  by  students  (/), 
classrooms  (y),  schools  (s),  and  fixed  site  effects  (d)\ 

Level  1 . Students . Yicsd  csd  T X icsdfi  T ® icsd 

(6)  Level  2:  Classrooms:  a0csd  = A00sd  + Wcsdn  + r]csd 

Level  3.  Schools . ^-oo sd  Yo  T Yi^sd  T SdGd  + Xsd8  + usd 

In  this  model,  YjCSd  is  an  outcome  measure  for  a student;  Clocui  is  a classroom-level  random  intercept;  Xoosi  is 
school-level  random  intercept;  are  student-level  baseline  covariates;  Wcsd  are  teacher-level  baseline 

covariates;  Ts,j  is  a binary  variable  equal  to  1 for  treatment  group  schools  and  0 for  control  group  schools;  Gj 
are  binary  variables  for  each  grantee;  Zsj  are  school-level  baseline  covariates11;  J3,  n,  yo,  Yu  Oj,  and  5 are 
parameter  vectors  to  be  estimated;  are  assumed  to  be  iid  N(0,  <T2r)  student-  or  teacher-level  random  error 
terms;  77a</are  iid  N(0,<J2,j)  classroom-specific  error  terms  that  are  assumed  to  be  distributed  independently  of 
the  Level  1 error  terms;  and  us,i  are  iid  (0,  O'2,,)  school-specific  error  terms  that  are  assumed  to  be  distributed 
independently  of  the  Level  1 and  2 error  terms. 

Inserting  the  Level  2 and  3 equations  into  the  Level  1 equation  yields  the  following  unified  model: 

(7)  Yicsd  — Yq  + Yi^sd  T ^d^d  T Zsd8  + WcsdTT  + XiCSdfL  + [ttscj  + hcsd  T 6icsd] 

In  this  formulation,  the  estimate  of  the  parameter,  y-i,  is  the  regression-adjusted,  multisite  impact  estimate. 
This  equation  is  similar  to  the  two-level  benchmark  model  in  equation  (1)  in  the  main  report  except  that  it 
includes  the  random  classroom  effect  tjcsj  in  the  error  term.  As  with  the  original  impact  estimates,  no 
significant  impacts  were  found  when  addressing  the  clustering  of  students  within  classroom  (table  B.22). 


11  In  practice,  school-level  covariates  were  not  used  in  estimation. 
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Table  B.22.  Combined-program  impacts  in  effect  size  units  when  accounting  for  classroom-level 
clustering 


Year  1 Year  2 Year  3 

(Spring  2005)  (Spring  2006)  (Spring  2007) 


p-value  of  p-value  of  p-value  of 


Scale-Report 

Impact 

impact 

Impact 

impact 

Impact 

impact 

Social  and  Emotional  Competence  Domain 

Self-Efficacy  for  Peer  Interactions-CR  (+) 

-0.02 

0.548 

-0.07A 

0.055 

-0.06 

0.114 

Normative  Beliefs  About  Aggression-CR  (-) 

-0.01 

0.796 

-0.01 

0.776 

0.00 

0.970 

Empathy-CR  (+) 

0.06 

0.186 

-0.02 

0.746 

-0.05 

0.254 

Behavior  Domain 

Altruistic  Behavior-CR  (+) 

-0.06 

0.115 

-0.06 

0.162 

-0.03 

0.408 

Altruistic  Behavior-PCR  (+) 

0.07A 

0.066 

-0.01 

0.871 

-0.02 

0.609 

Altruistic  Behavior-TRS  (+) 

0.1 5A 

0.077 

0.05 

0.545 

-0.03 

0.671 

Positive  Social  Behavior-PCR  (+) 

-0.01 

0.632 

0.04 

0.264 

0.02 

0.526 

Positive  Social  Behavior-TRS  (+) 

0.01 

0.703 

0.01 

0.849 

0.05 

0.400 

Problem  Behavior-CR  (-) 

0.00 

0.993 

0.00 

0.940 

0.05 

0.299 

Problem  Behavior-PCR  (+) 

0.01 

0.876 

-0.03 

0.336 

-0.03 

0.435 

Problem  Behavior-TRS  (+) 

0.03 

0.506 

-0.03 

0.598 

-0.02 

0.678 

ADHD-Related  Behavior-TRS  (-) 

0.00 

0.935 

-0.05 

0.344 

-0.03 

0.492 

Academics  Domain 

Engagement  with  Learning-CR  (+) 

-0.04 

0.240 

-0.02 

0.643 

-0.03 

0.509 

Academic  Competence  and 

Motivation-TRS  (+) 

-0.01 

0.614 

-0.01 

0.772 

0.00 

0.928 

Perceptions  of  School  Climate  Domain 

Positive  School  Orientation-CR  (+) 

0.03 

0.554 

0.04 

0.611 

-0.08 

0.310 

Negative  School  Orientation-CR  (-) 

-0.05 

0.180 

-0.04 

0.451 

-0.01 

0.885 

Student  Afraid  at  School-CR  (-) 

-0.06 

0.144 

-0.08 

0.101 

-0.01 

0.867 

Victimization  at  School-CR  (-) 

-0.02 

0.661 

-0.05 

0.321 

0.00 

0.958 

A Significantly  different  from  zero  at  the  .10  to  > .05  level. 

1 1mpacts  are  in  effect  size  (standard  deviation)  units  and  were  calculated  by  dividing  the  estimated  impact  by  the  standard  deviation 
of  the  outcome  measure  for  the  control  group. 

2 The  p-value  is  from  a two-tailed  t test  to  gauge  the  statistical  significance  of  the  impact  estimate. 

NOTE:  Abbreviations  are 
CR:  Child  Report 
PCR:  Primary  Caregiver  Report 
TRS:  Teacher  Report  on  Student 
ADHD:  Attention  deficit  hyperactivity  disorder 

The  +/-  signs  in  parentheses  indicate  the  direction  of  a beneficial  outcome.  No  findings  were  found  statistically  significant  below  the 
.05  level.  All  impact  estimates  were  calculated  using  regression  models,  where  each  program  and  school  within  a program  was 
weighted  equally.  The  standard  errors  of  all  estimates  account  for  design  effects  due  to  unequal  weighting  and  the  clustering  of 
students  within  schools. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 


B-46 


Appendix  B:  Technical  Notes 


Accounting  for  the  Pairwise  Matching  of  Schools 

Under  an  experimental  design  where  the  random  assignment  of  units  is  conducted  within  strata,  it  is 
customary  to  account  for  the  stratified  design  in  the  analysis  by  including  binary  variables  of  the  strata  as 
covariates  in  the  regression  models.  Under  a clustered  design,  the  inclusion  of  these  variables  will  reduce 
design  effects  to  the  extent  that  they  are  correlated  with  the  outcome  measures.  Stated  differently,  under 
stratified  designs,  intraclass  correlations  (ICCs)  pertain  to  design  effects  within  strata.  Thus,  stratified  sampling 
will  reduce  ICCs  if  mean  school  outcomes  tend  to  be  more  similar  within  strata  than  across  strata. 

This  procedure,  however,  cannot  be  used  for  a design  where  random  assignment  is  conducted  for  units 
within  matched  pairs.  This  is  because  when  pair  binary  variables  and  pair-by-treatment  interaction  terms  are 
included  as  model  covariates,  the  school-level  variance  term  represents  the  extent  to  which  mean  school 
outcomes  vary  within  pairs  and  within  a research  group  (Murray  1998).  Thus,  because  there  is  only  one 
treatment  and  one  control  group  unit  per  pair,  there  are  not  enough  degrees  of  freedom  to  estimate  the 
within-pair  variance  terms. 

To  account  for  this  problem,  the  pairs  were  treated  as  another  HLM  level.  Specifically,  the  following  three- 
level  HLM  model  was  estimated,  where  pairs  were  indexed  by  p: 

Level  1:  Students /Teachers:  Yispd  = a0spd  + Xispd(3  + eispd 
(8)  Level  2:  Schools:  cc^ spd  — Yopd  + YipdTspd  T ^d^d  T Xspd8  + uspd 

Level  3:  Pairs:  Yopd  4 f 0pd<  Yipd  ^-i  T fpd 

In  this  model,  (/)Pd  and  Tpj  are  independent  and  identically  distributed  N(0,  cf  p and  independent  and  identically 
distributed  N(0,&  T)  pair-level  random  error  terms,  respectively.  Recursively  inserting  the  Level  2 and  3 
equations  into  the  Level  1 equation  yields  the  following  unified  model: 

(9)  Yispd  ~ ^-0  ^-1  Y'spd  ^d^d  ^spd^  ^ispdP  [Tp^spd  $p  uspd  eispd] 

Under  this  model,  the  key  estimatable  component  of  the  variance  expression  for  the  impact  estimate  is 
(c/  JP),  where  P is  the  total  number  of  pairs.  This  variance  component  signifies  the  extent  to  which  wipacts 
vary  across  pairs.  In  contrast,  the  leading  term  in  the  variance  expression  for  the  benchmark  approach  is 
( 2cf„/P ),  which  signifies  the  extent  to  which  mean  outcomes  vary  across  schools. 

It  is  unclear  a priori  whether  the  pairwise  matching  model  will  produce  more  precise  impact  estimates  than 
the  original  approach.  On  the  one  hand,  precision  gains  could  occur  if  cf  T < 2& „ which  will  likely  be  the  case 
as  long  as  there  is  some  correlation  between  mean  outcomes  for  the  treatment  and  control  group  schools 
within  the  same  pairs  (that  is,  if  the  matching  was  somewhat  “successful”).  On  the  other  hand,  precision 
losses  will  occur  due  to  a reduction  in  the  number  of  degrees  of  freedom  available  for  analysis,  because  there 
are  only  half  as  many  pairs  as  there  are  schools.  Thus,  it  is  an  empirical  issue  as  to  which  approach  will  yield 
more  precise  impact  estimates. 

Accounting  for  pairwise  matching  in  the  analysis  did  not  change  the  statistical  significance  of  the  estimated 
impacts  with  one  exception:  a detrimental  impact  was  found  for  Altruistic  Behavior  (CR)  in  Year  1 (table 
B.23). 
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Table  B.23.  Combined-program  impacts  in  effect  size  units  when  accounting  for  pairwise 
matching 


Year  1 Year  2 Year  3 

(Spring  2005)  (Spring  2006)  (Spring  2007) 


p-value  of  p-value  of  p-value  of 


Scale-Report 

Impact 

impact 

Impact 

impact 

Impact 

impact 

Social  and  Emotional  Competence  Domain 

Self-Efficacy  for  Peer  Interactions-CR  (+) 

-0.03 

0.386 

-0.07 

0.135 

-0.05 

0.217 

Normative  Beliefs  About  Aggression-CR  (-) 

0.00 

0.951 

0.00 

0.957 

0.01 

0.876 

Empathy-CR  (+) 

0.05 

0.302 

-0.02 

0.742 

-0.05 

0.343 

Behavior  Domain 

Altruistic  Behavior-CR  (+) 

-0.07* 

0.036 

-0.03 

0.444 

-0.03 

0.477 

Altruistic  Behavior-PCR  (+) 

0.05 

0.118 

-0.03 

0.420 

-0.06 

0.157 

Altruistic  Behavior-TRS  (+) 

0.08 

0.374 

0.02 

0.910 

-0.02 

0.868 

Positive  Social  Behavior-PCR  (+) 

-0.02 

0.528 

0.01 

0.772 

0.02 

0.606 

Positive  Social  Behavior-TRS  (+) 

0.01 

0.709 

-0.03 

0.618 

0.05 

0.470 

Problem  Behavior-CR  (-) 

0.01 

0.797 

-0.01 

0.906 

0.02 

0.759 

Problem  Behavior-PCR  (+) 

-0.01 

0.814 

-0.02 

0.602 

-0.05 

0.371 

Problem  Behavior-TRS  (+) 

0.03 

0.444 

-0.02 

0.720 

-0.05 

0.405 

ADHD-Related  Behavior-TRS  (-) 

0.02 

0.648 

-0.02 

0.712 

-0.09 

0.105 

Academics  Domain 

Engagement  with  Learning-CR  (+) 

-0.05A 

0.083 

-0.03 

0.508 

-0.06 

0.201 

Academic  Competence  and 

Motivation-TRS  (+) 

-0.03 

0.345 

-0.04 

0.338 

-0.02 

0.718 

Perceptions  of  School  Climate  Domain 

Positive  School  Orientation-CR  (+) 

0.01 

0.852 

0.04 

0.395 

-0.05 

0.424 

Negative  School  Orientation-CR  (-) 

-0.03 

0.303 

-0.07 

0.192 

-0.03 

0.543 

Student  Afraid  at  School-CR  (-) 

-0.06A 

0.089 

-0.05 

0.328 

0.04 

0.440 

Victimization  at  School-CR  (-) 

0.00 

0.999 

-0.03 

0.370 

0.01 

0.791 

* Significantly  different  from  zero  at  the  .05  level. 

A Significantly  different  from  zero  at  the  .10  to  > .05  level. 

1 1mpacts  are  in  effect  size  (standard  deviation)  units  and  were  calculated  by  dividing  the  estimated  impact  by  the  standard  deviation 
of  the  outcome  measure  for  the  control  group. 

2 The  p-value  is  from  a two-tailed  t test  to  gauge  the  statistical  significance  of  the  impact  estimate. 

NOTE:  Abbreviations  are 
CR:  Child  Report 
PCR:  Primary  Caregiver  Report 
TRS:  Teacher  Report  on  Student 
ADHD:  Attention  deficit  hyperactivity  disorder 

The  +/-  signs  in  parentheses  indicate  the  direction  of  a beneficial  outcome.  All  impact  estimates  were  calculated  using  regression 
models,  where  each  program  and  school  within  a program  was  weighted  equally.  The  standard  errors  of  all  estimates  account  for 
design  effects  due  to  unequal  weighting  and  the  clustering  of  students  within  schools. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Treating  Pretest  Measures  as  Dependent  Variables 

In  the  original  model,  initial  measures  of  the  outcomes  were  included  as  covariates  in  the  regression  models. 
An  alternative  approach  was  to  treat  these  pretests  as  dependent  variables.  This  approach  was  implemented  in 
two  ways  using  Year  1 and  Year  2 data.  First,  models  were  estimated  where  the  dependent  variable  was 
defined  as  the  difference  between  posttest  and  pretest  scores  (that  is,  gain  scores),  which  yields  a difference-in- 
difference estimator.12  Second,  the  pretest  and  posttest  scores  were  stacked  as  separate  observations  and  the 
model  included  time-by-treatment  interaction  terms  as  covariates  as  shown  here: 

(10)  Yisd  = y0+  nTsd  + YiPost  + y3Post  * Tsd  + Xisd(3  + 0d+  ZsdS  + [usd  + Post  * rjsd  + eisd ] 

where 


• is  an  outcome  measure  for  student  (or  teacher)  i; 

• Xisd  are  student -level  (or  teacher-level)  baseline  covariates; 

• Tsd  is  a binary  variable  equal  to  1 for  treatment  group  schools  and  0 for  control  group  schools; 

• dd  are  site-specific  fixed  effects; 

• Z„/are  school-level  baseline  covariates;13 

• Post  is  a binary  variable  that  equals  1 for  the  posttest  observations  and  0 for  the  pretest  observations; 

• Yo,  Yb  Y?>  Yh  and  & are  fixed  parameter  vectors  to  be  estimated;  and 

• eud  are  independent  and  identically  distributed  N(0,  <J2e)  student-level  (or  teacher-level)  random  error 
terms. 

Furthermore,  usd  and  t]sd  are  independent  and  identically  distributed  N(0,  <J2„)  and  A 1(0,  ( 7 2,J  school-level  random 
error  terms,  respectively,  that  capture  the  correlations  between  the  outcomes  of  students  in  the  same  schools 
and  time  periods  and  that  are  distributed  independently  of  the  student-level  (or  teacher-level)  error  terms.  In 
this  formulation,  the  estimate  of  y)  is  the  difference-in-difference  impact  estimate,  and  the  leading  term  in  the 
variance  expression  for  the  impact  estimate  is  a function  of  O 2n,  which  signifies  the  extent  to  which  gain 
scores  vary  across  schools  within  programs. 

For  each  estimation  approach,  the  new  entrants  were  excluded  from  the  analysis  sample  because  of  missing 
pretest  scores.14  Thus,  the  models  were  estimated  using  only  original  cohort  members. 

In  the  original  model,  none  of  the  36  estimated  impacts  was  found  to  be  significant.  The  treatment  of  the 
pretests  as  dependent  variables  rather  than  as  covariates  gave  similar  results  with  two  exceptions.  When  using 
the  gain  score  as  the  dependent  variable,  a statistically  significant  beneficial  impact  was  found  for  Negative 
School  Orientation  (in  Year  2 with  an  effect  size  of  -0.15)  and  Student  Afraid  at  School  (in  Year  1 with  an 
effect  size  of  0.09)  (table  B.24).  When  using  pretests  and  posttests  as  dependent  variables,  a beneficial  impact 
on  Negative  School  Orientation  was  found  in  Year  2 (effect  size  of  -0.16)  and  on  Student  Afraid  at  School  in 
Year  1 (effect  size  of  0.10)  (table  B.25). 


12  In  these  models,  the  pretests  were  excluded  from  the  model  covariates,  although  other  covariates  remained  the  same 
as  in  the  benchmark  models. 

13  For  teacher-level  outcomes,  the  Z variables  included  teacher-level  baseline  covariates.  Although  the  initial  model 
specified  the  use  of  school-level  covariates,  none  were  used  in  the  empirical  estimation. 

14  Using  the  mean-imputed  pretests  for  the  new  entrants  would  yield  standard  errors  that  are  biased  downward. 
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Table  B.24.  Combined-program  impacts  in  effect  size  units  when  dependent  variable  is  the 
pretest-posttest  difference  (gain  score) 


Scale-Report 

Year  1 

(Spring  2005) 

p-value  of 
Impact1  impact2 

Year  2 

(Spring  2006) 

p-value  of 
Impact1  impact2 

Social  and  Emotional  Competence  Domain 

Self-Efficacy  for  Peer  Interactions-CR  (+) 

0.01 

0.883 

-0.04 

0.342 

Normative  Beliefs  About  Aggression-CR  (-) 

0.02 

0.628 

0.01 

0.864 

Empathy-CR  (+) 

0.03 

0.663 

-0.01 

0.805 

Behavior  Domain 

Altruistic  Behavior-CR  (+) 

-0.09A 

0.057 

-0.04 

0.482 

Altruistic  Behavior-PCR  (+) 

0.04 

0.356 

-0.04 

0.430 

Altruistic  Behavior-TRS  (+) 

0.08 

0.585 

-0.08 

0.603 

Positive  Social  Behavior-PCR  (+) 

-0.02 

0.569 

0.03 

0.546 

Positive  Social  Behavior-TRS  (+) 

-0.03 

0.599 

-0.08 

0.360 

Problem  Behavior-CR  (-) 

0.03 

0.605 

-0.03 

0.587 

Problem  Behavior-PCR  (+) 

0.01 

0.882 

0.01 

0.872 

Problem  Behavior-TRS  (+) 

0.08 

0.249 

0.00 

0.998 

ADHD-Related  Behavior-TRS  (-) 

0.06 

0.320 

0.00 

0.989 

Academics  Domain 

Engagement  with  Learning-CR  (+) 

-0.03 

0.444 

0.03 

0.539 

Academic  Competence  and  Motivation-TRS  (+) 

-0.04 

0.470 

-0.04 

0.540 

Perceptions  of  School  Climate  Domain 

Positive  School  Orientation-CR  (+) 

0.00 

0.989 

0.07 

0.239 

Negative  School  Orientation-CR  (-) 

-0.06 

0.126 

-0.15* 

0.007 

Student  Afraid  at  School-CR  (-) 

-0.09* 

0.028 

-0.04 

0.433 

Victimization  at  School-CR  (-) 

0.00 

0.973 

-0.02 

0.639 

* Significantly  different  from  zero  at  the  .05  level. 

A Significantly  different  from  zero  at  the  .10  to  > .05  level. 

1 1mpacts  are  in  effect  size  (standard  deviation)  units  and  were  calculated  by  dividing  the  estimated  impact  by  the  standard  deviation 
of  the  outcome  measure  for  the  control  group. 

2 The  p-value  is  from  a two-tailed  t test  to  gauge  the  statistical  significance  of  the  impact  estimate. 

NOTE:  Abbreviations  are 
CR:  Child  Report 
PCR:  Primary  Caregiver  Report 
TRS:  Teacher  Report  on  Student 
ADHD:  Attention  deficit  hyperactivity  disorder 

The  +/-  signs  in  parentheses  indicate  the  direction  of  a beneficial  outcome.  All  impact  estimates  were  calculated  using  regression 
models,  where  each  program  and  school  within  a program  was  weighted  equally.  The  standard  errors  of  all  estimates  account  for 
design  effects  due  to  unequal  weighting  and  the  clustering  of  students  within  schools. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Table  B.25.  Combined-program  impacts  in  effect  size  units  when  posttest  and  pretest  are  treated 
as  separate  dependent  variables 


Year  1 

Year  2 

(Spring  2005) 

(Spring  2006) 

Scale-Report 

Impact1 

p-value  of 
impact2 

Impact1 

p-value  of 
impact2 

Social  and  Emotional  Competence  Domain 

Self-Efficacy  for  Peer  Interactions-CR  (+) 

0.01 

0.818 

-0.04 

0.417 

Normative  Beliefs  About  Aggression-CR  (-) 

0.03 

0.486 

0.02 

0.799 

Empathy-CR  (+) 

0.02 

0.708 

-0.03 

0.654 

Behavior  Domain 

Altruistic  Behavior-CR  (+) 

-0.09A 

0.062 

-0.07 

0.341 

Altruistic  Behavior-PCR  (+) 

0.03 

0.567 

-0.04 

0.461 

Altruistic  Behavior-TRS  (+) 

0.07 

0.607 

-0.09 

0.627 

Positive  Social  Behavior-PCR  (+) 

-0.03 

0.544 

0.02 

0.598 

Positive  Social  Behavior-TRS  (+) 

-0.02 

0.645 

-0.08 

0.369 

Problem  Behavior-CR  (-) 

0.03 

0.521 

-0.02 

0.786 

Problem  Behavior-PCR  (+) 

0.01 

0.785 

0.02 

0.715 

Problem  Behavior-TRS  (+) 

0.04 

0.345 

0.01 

0.912 

ADHD-Related  Behavior-TRS  (-) 

0.04 

0.390 

0.00 

0.988 

Academics  Domain 

Engagement  with  Learning-CR  (+) 

-0.05 

0.419 

0.01 

0.843 

Academic  Competence  and  Motivation-TRS  (+) 

-0.01 

0.683 

-0.03 

0.535 

Perceptions  of  School  Climate  Domain 

Positive  School  Orientation-CR  (+) 

-0.01 

0.827 

0.07 

0.311 

Negative  School  Orientation-CR  (-) 

-0.06 

0.238 

-0.16* 

0.024 

Student  Afraid  at  School-CR  (-) 

-0.10* 

0.043 

-0.05 

0.394 

Victimization  at  School-CR  (-) 

0.00 

0.944 

-0.02 

0.692 

* Significantly  different  from  zero  at  the  .05  level. 

A Significantly  different  from  zero  at  the  .10  to  > .05  level. 

1 1mpacts  are  in  effect  size  (standard  deviation)  units  and  were  calculated  by  dividing  the  estimated  impact  by  the  standard  deviation 
of  the  outcome  measure  for  the  control  group. 

2 The  p-value  is  from  a two-tailed  t test  to  gauge  the  statistical  significance  of  the  impact  estimate. 

NOTE:  Abbreviations  are 
CR:  Child  Report 
PCR:  Primary  Caregiver  Report 
TRS:  Teacher  Report  on  Student 
ADHD:  Attention  deficit  hyperactivity  disorder 

The  +/-  signs  in  parentheses  indicate  the  direction  of  a beneficial  outcome.  All  impact  estimates  were  calculated  using  regression 
models,  where  each  program  and  school  within  a program  was  weighted  equally.  The  standard  errors  of  all  estimates  account  for 
design  effects  due  to  unequal  weighting  and  the  clustering  of  students  within  schools. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Using  Alternative  Software  Packages 

The  original  models  were  estimated  using  the  PROC  MIXED  procedure  in  SAS.  To  check  the  sensitivity  of 
the  results,  the  impacts  were  estimated  using  alternative  statistical  packages — SUDAAN  PROC  REGRESS 
and  HLM6 — that  account  for  clustering  effects  in  slightly  different  ways  than  SAS  PROC  MIXED.  SAS 
PROC  MIXED  uses  a maximum-likelihood  approach  to  general  linear  mixed  models,  whereas  the  SUDAAN 
procedure  is  based  on  the  Taylor-series  linearization  method,  combined  with  variance  estimation  formulas 
specific  to  the  sample  design.  Similar  to  SAS  PROC  MIXED,  HLM6  uses  a restricted  maximum  likelihood 
approach  to  estimate  the  parameters,  but  HLM6  uses  the  Expectation-Maximization  algorithm  to  obtain  the 
estimates,  whereas  SAS  PROC  MIXED  uses  a ridge-stabilized  Newton- Raphson  algorithm. 

The  three  statistical  packages  produced  similar  impact  and  standard  error  estimates  (table  B.26)  with  one 
exception.  When  SUDAAN  was  used,  the  estimate  for  Altruistic  Behavior  (Primary  Caregiver  Report) 
showed  a beneficial  statistically  significant  effect  in  Year  1 (with  an  effect  size  of  0.07). 
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Table  B.26.  Combined-program  impacts  in  effect  size  units  when  using  different  statistical  software  packages 


Year  1 

Year  2 

(Spring  2005) 

(Spring  2006) 

SUDAAN  HLM6 

SUDAAN  HLM6 

p-value  of  p-value  of  p-value  of  p-value  of 


Scale-Report Impact1 impact2  Impact1 impact2  Impact1 impact2  Impact1  impact2 


Social  and  Emotional  Competence  Domain 

Self-Efficacy  for  Peer  Interactions-CR  (+) 

-0.02 

0.518 

-0.04 

0.291 

-0.07A 

0.084 

-0.06 

0.209 

Normative  Beliefs  About  Aggression-CR  (-) 

-0.01 

0.875 

0.01 

0.896 

-0.03 

0.603 

0.03 

0.624 

Empathy-CR  (+) 

0.06 

0.148 

0.05 

0.223 

-0.02 

0.729 

-0.01 

0.779 

Behavior  Domain 

Altruistic  Behavior-CR  (+) 

-0.06A 

0.092 

-0.06 

0.105 

-0.06 

0.144 

-0.03 

0.430 

Altruistic  Behavior-PCR  (+) 

0.07* 

0.037 

0.05 

0.213 

0.00 

0.963 

-0.02 

0.605 

Altruistic  Behavior-TRS  (+) 

0.08 

0.288 

0.08 

0.480 

0.06 

0.629 

-0.09 

0.617 

Positive  Social  Behavior-PCR  (+) 

-0.01 

0.707 

0.00 

0.901 

0.04 

0.178 

0.01 

0.787 

Positive  Social  Behavior-TRS  (+) 

0.02 

0.625 

0.01 

0.812 

-0.02 

0.728 

-0.03 

0.691 

Problem  Behavior-CR  (-) 

0.01 

0.872 

0.04 

0.436 

-0.01 

0.893 

0.02 

0.756 

Problem  Behavior-PCR  (+) 

0.00 

0.940 

-0.04 

0.308 

-0.03 

0.300 

-0.02 

0.633 

Problem  Behavior-TRS  (+) 

0.02 

0.572 

0.02 

0.662 

-0.01 

0.800 

-0.03 

0.543 

ADHD-Related  Behavior-TRS  (-) 

0.00 

0.942 

0.02 

0.696 

-0.02 

0.497 

-0.02 

0.768 

Academics  Domain 

Engagement  with  Learning-CR  (+) 

-0.04 

0.182 

-0.07A 

0.053 

-0.03 

0.400 

-0.03 

0.521 

Academic  Competence  and  Motivation-TRS  (+) 

-0.02 

0.365 

-0.08A 

0.053 

-0.02 

0.498 

-0.05 

0.282 

See  notes  at  end  of  table. 
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Table  B.26.  Combined-program  impacts  in  effect  size  units  when  using  different  statistical  software  packages — Continued 


Year  1 Year  2 

(Spring  2005) (Spring  2006) 

SUDAAN  HLM6  SUDAAN  HLM6 


p-value  of  p-value  of  p-value  of  p-value  of 


Scale-Report 

Impact1 

1 

impact 

Impact1 

impact2 

Impact1 

impact2 

Impact1 

impact2 

Perceptions  of  School  Climate  Domain 
Positive  School  Orientation-CR  (+) 

0.03 

0.467 

0.00 

0.973 

0.05 

0.432 

0.03 

0.703 

Negative  School  Orientation-CR  (-) 

-0.05 

0.127 

-0.01 

0.869 

-0.04 

0.344 

-0.06 

0.364 

Student  Afraid  at  School-CR  (-) 

-0.06 

0.127 

-0.04 

0.391 

-0.06 

0.166 

-0.04 

0.500 

Victimization  at  School-CR  (-) 

-0.01 

0.765 

0.02 

0.648 

-0.04 

0.370 

-0.03 

0.506 

* Significantly  different  from  zero  at  the  .05  level. 

A Significantly  different  from  zero  at  the  .10  to  > .05  level. 

1 1mpacts  are  in  effect  size  (standard  deviation)  units  and  were  calculated  by  dividing  the  estimated  impact  by  the  standard  deviation  of  the  outcome  measure  for  the  control  group. 
2 The  p-value  is  from  a two-tailed  t test  to  gauge  the  statistical  significance  of  the  impact  estimate. 

NOTE:  Abbreviations  are 
CR:  Child  Report 
PCR:  Primary  Caregiver  Report 
TRS:  Teacher  Report  on  Student 
ADHD:  Attention  deficit  hyperactivity  disorder 

The  +/-  signs  in  parentheses  indicate  the  direction  of  a beneficial  outcome.  All  impact  estimates  were  calculated  using  regression  models,  where  each  program  and  school  within  a 
program  was  weighted  equally.  The  standard  errors  of  all  estimates  account  for  design  effects  due  to  unequal  weighting  and  the  clustering  of  students  within  schools. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Imputing  Missing  Outcomes 

In  the  original  models,  missing  outcome  data  were  not  imputed  for  (1)  study  nonconsenters,  (2)  survey 
nonrespondents,  and  (3)  survey  respondents  who  did  not  provide  enough  information  to  construct  the 
outcomes.  Instead,  adjustments  for  missing  outcome  data  were  made  using  sample  weights  so  that  the  impact 
estimates  could  be  generalized  to  the  full  sample  universe  (and  not  just  to  survey  respondents). 

As  a sensitivity  test,  models  were  estimated  with  imputed  outcome  data.  Nearly  all  students  with  missing  data 
were  study  nonconsenters  for  whom  baseline  data  were  not  available.  Thus,  imputation  methods  that  used 
covariates  as  predictor  variables  could  not  be  used.  Instead,  missing  outcome  data  were  imputed  using  a 
hotdeck  imputation  procedure  (Rubin  1987),  where  a missing  value  for  a child  was  replaced  by  a nonmissing 
value  for  a randomly  selected  child  in  the  same  classroom.  Five  imputed  data  sets  were  generated  and 
estimated  impacts  (/,)  and  variances  (I /)  were  made  for  each  one.  The  overall  impact  estimates  were 
calculated  as  the  average  of  the  five  impact  estimates.  Variances  of  the  estimated  impacts  were  obtained  using 
the  following  formula: 

(11)  V + (l  + —)b 
\ mJ 

where  V is  the  average  of  the  Tis  across  the  five  data  sets,  m — 5 is  the  number  of  imputed  data  sets,  and  B is 
the  following  between-imputation  variance  estimate: 

m 

(12)  b = — y Oi  - w 

m — 1 Z_i 

i=i 

Because  child-level  covariates  were  not  available  for  study  nonconsenters,  the  regression  models  were 
estimated  without  covariates,  except  for  program-level  binary  variables. 

As  with  the  original  impact  estimates,  no  significant  impacts  were  found  when  using  the  multiple  imputation 
approach  (table  B.27). 
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Table  B.27.  Combined-program  impacts  in  effect  size  units  when  using  imputation  procedures 
for  missing  outcome  data 


Scale-Report 

Year  1 

(Spring  2005) 

p-value  of 
Impact1  impact2 

Year  2 

(Spring  2006) 

p-value  of 
Impact1  impact2 

Social  and  Emotional  Competence  Domain 

Self-Efficacy  for  Peer  Interactions-CR  (+) 

-0.05 

0.293 

-0.06 

0.296 

Normative  Beliefs  About  Aggression-CR  (-) 

-0.01 

0.883 

-0.02 

0.733 

Empathy-CR  (+) 

0.06 

0.375 

-0.04 

0.543 

Behavior  Domain 

Altruistic  Behavior-CR  (+) 

-0.06 

0.307 

-0.07 

0.274 

Altruistic  Behavior-PCR  (+) 

0.06 

0.305 

-0.01 

0.831 

Altruistic  Behavior-TRS  (+) 

0.12 

0.278 

0.05 

0.722 

Positive  Social  Behavior-PCR  (+) 

-0.01 

0.891 

0.01 

0.859 

Positive  Social  Behavior-TRS  (+) 

0.01 

0.868 

-0.06 

0.530 

Problem  Behavior-CR  (-) 

0.03 

0.697 

0.02 

0.803 

Problem  Behavior-PCR  (+) 

-0.05 

0.428 

-0.06 

0.248 

Problem  Behavior-TRS  (+) 

0.03 

0.724 

0.01 

0.913 

ADHD-Related  Behavior-TRS  (-) 

0.01 

0.831 

-0.02 

0.743 

Academics  Domain 

Engagement  with  Learning-CR  (+) 

-0.06 

0.236 

-0.05 

0.265 

Academic  Competence  and  Motivation-TRS  (+) 

-0.05 

0.440 

-0.08 

0.211 

Perceptions  of  School  Climate  Domain 

Positive  School  Orientation-CR  (+) 

0.00 

0.981 

0.02 

0.821 

Negative  School  Orientation-CR  (-) 

-0.02 

0.788 

0.00 

0.954 

Student  Afraid  at  School-CR  (-) 

-0.04 

0.549 

-0.05 

0.511 

Victimization  at  School-CR  (-) 

-0.02 

0.661 

-0.06 

0.279 

1 1mpacts  are  in  effect  size  (standard  deviation)  units  and  were  calculated  by  dividing  the  estimated  impact  by  the  standard  deviation 
of  the  outcome  measure  for  the  control  group. 

2 The  p-value  is  from  a two-tailed  t test  to  gauge  the  statistical  significance  of  the  impact  estimate. 

NOTE:  Abbreviations  are 
CR:  Child  Report 
PCR:  Primary  Caregiver  Report 
TRS:  Teacher  Report  on  Student 
ADHD:  Attention  deficit  hyperactivity  disorder 

The  +/-  signs  in  parentheses  indicate  the  direction  of  a beneficial  outcome.  No  findings  were  found  statistically  significant  at  or  below 
the  .05  level.  All  impact  estimates  were  calculated  using  regression  models,  where  each  program  and  school  within  a program  was 
weighted  equally.  The  standard  errors  of  all  estimates  account  for  design  effects  due  to  unequal  weighting  and  the  clustering  of 
students  within  schools. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Averaging  Program-Level  Impact  Estimates 

An  alternative  approach  for  estimating  combined-program  impacts  was  to  calculate  the  simple  average  of  the 
seven  program-specific  impact  estimates.  This  procedure  was  done  in  two  ways.  First,  the  seven  program-by- 
treatment interaction  terms  (that  is,  0p*Tsp  terms)  were  added  as  covariates  to  the  original  combined-program 
model,  and  the  treatment  status  binary  variable  was  excluded  from  the  model.  In  this  formulation,  the 
parameter  estimate  on  a specific  program-by-treatment  interaction  term  was  the  impact  estimate  for  that 
program.  Using  this  approach,  the  combined-program  impact  estimate  was  calculated  as  the  simple  average  of 
the  parameter  estimates  on  the  seven  interaction  terms,  and  the  associated  standard  error  (SE)  was  calculated 
using  the  following  formula: 


1 

(13)  S E (site  average)  = — 

where  Vpis  the  estimated  variance  of  the  impact  in  program  p and  Qk  is  the  estimated  covariance  between  the 
impacts  in  programs  j and  k. 

In  the  second  approach,  combined-program  impact  estimates  were  calculated  by  averaging  the  impact 
estimates  from  the  HLM  models  that  were  estimated  separately  by  program.  Equation  (13)  was  used  to 
calculate  the  standard  errors  with  Cjk  set  to  0 (because  observations  were  independent  across  programs).  This 
approach  differed  slightly  from  the  first  approach  for  several  reasons.  First,  a different  set  of  baseline 
covariates  was  used  for  each  program-specific  model,  whereas  a common  covariate  set  was  used  in  the 
pooled  model.  Second,  in  the  pooled  model,  a single  (average)  clustering  effect  was  applied  to  all  programs, 
whereas,  in  the  program-specific  models,  the  clustering  effects  were  allowed  to  vary  across  programs. 

As  table  B.28  shows,  the  combined -program  impact  estimates  from  these  two  approaches  in  which  program- 
specific  impact  estimates  were  calculated  directly  and  then  averaged  were  very  similar  to  the  original  results 
that  relied  on  weights  to  give  each  program  equal  weight  in  the  analysis.  The  only  difference  was  that  a 
beneficial  significant  impact  (effect  size  of  0.07)  was  found  for  Positive  Social  Behavior  (PCR)  in  Year  2 when 
using  the  average  of  the  estimates  from  the  seven  program-level  models. 


7 7 


p= 1 j=l k>] 


Cjk 


B-57 


B-58 


Table  B.28.  Combined-program  impacts  in  effect  size  units  using  averages  of  program-specific  impacts 


Year  1 

(Spring  2005) 

Year  2 

(Spring  2006) 

Program  by 

Seven  separate 

Program  by 

Seven  separate 

treatment 

estimates 

treatment 

estimates 

p-value  of  p-value  of  p-value  of  p-value  of 


Scale-Report Impact1 impact2  Impact1  impact2  Impact1  impact2  Impact1  impact2 


Social  and  Emotional  Competence  Domain 

Self-Efficacy  for  Peer  Interactions-CR  (+) 

-0.02 

0.543 

-0.01 

0.717 

-0.07 

0.107 

-0.06 

0.145 

Normative  Beliefs  About  Aggression-CR  (-) 

-0.01 

0.871 

-0.01 

0.754 

-0.02 

0.748 

-0.01 

0.535 

Empathy-CR  (+) 

0.06 

0.176 

0.07 

0.150 

-0.02 

0.721 

-0.02 

0.596 

Behavior  Domain 

Altruistic  Behavior-CR  (+) 

-0.06 

0.107 

-0.07A 

0.057 

-0.06 

0.193 

-0.08 

0.100 

Altruistic  Behavior-PCR  (+) 

0.07A 

0.058 

0.07A 

0.095 

0.00 

0.965 

-0.01 

0.810 

Altruistic  Behavior-TRS  (+) 

0.08 

0.378 

0.07 

0.460 

0.02 

0.902 

0.06 

0.868 

Positive  Social  Behavior-PCR  (+) 

-0.01 

0.725 

-0.01 

0.774 

0.05 

0.177 

0.07* 

0.037 

Positive  Social  Behavior-TRS  (+) 

0.01 

0.709 

0.01 

0.860 

-0.02 

0.768 

0.00 

0.957 

Problem  Behavior-CR  (-) 

0.01 

0.892 

0.01 

0.886 

0.00 

0.998 

0.02 

0.879 

Problem  Behavior-PCR  (+) 

0.00 

0.930 

0.01 

0.811 

-0.03 

0.341 

-0.04 

0.271 

Problem  Behavior-TRS  (+) 

0.02 

0.607 

0.03 

0.545 

-0.01 

0.782 

-0.02 

0.513 

ADHD-Related  Behavior-TRS  (-) 

0.00 

0.958 

0.00 

0.970 

-0.03 

0.508 

-0.05 

0.416 

Academics  Domain 

Engagement  with  Learning-CR  (+) 

-0.04 

0.217 

-0.04 

0.345 

-0.03 

0.490 

-0.03 

0.555 

Academic  Competence  and  Motivation-TRS  (+) 

-0.02 

0.433 

-0.02 

0.448 

-0.02 

0.653 

0.01 

0.835 

See  notes  at  end  of  table. 
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Table  B.28.  Combined-program  impacts  in  effect  size  units  using  averages  of  program-specific  impacts — Continued 


Year  1 Year  2 

(Spring  2005) (Spring  2006) 

Program  by  Seven  separate  Program  by  Seven  separate 

treatment  estimates  treatment  estimates 


p-value  of  p-value  of  p-value  of  p-value  of 


Scale-Report 

Impact1 

impact2 

Impact1 

impact2 

Impact1 

impact2 

Impact1 

impact2 

Perceptions  of  School  Climate  Domain 
Positive  School  Orientation-CR  (+) 

0.03 

0.586 

0.04 

0.438 

0.03 

0.735 

0.03 

0.695 

Negative  School  Orientation-CR  (-) 

-0.05 

0.167 

-0.05 

0.209 

-0.03 

0.503 

-0.02 

0.653 

Student  Afraid  at  School-CR  (-) 

-0.06 

0.171 

-0.06 

0.140 

-0.06 

0.218 

-0.09 

0.101 

Victimization  at  School-CR  (-) 

-0.01 

0.793 

-0.01 

0.763 

-0.04 

0.431 

-0.02 

0.639 

* Significantly  different  from  zero  at  the  .05  level. 

A Significantly  different  from  zero  at  the  .10  to  > .05  level. 

1 1mpacts  are  in  effect  size  (standard  deviation)  units  and  were  calculated  by  dividing  the  estimated  impact  by  the  standard  deviation  of  the  outcome  measure  for  the  control  group. 
2 The  p-value  is  from  a two-tailed  t test  to  gauge  the  statistical  significance  of  the  impact  estimate. 

NOTE:  Abbreviations  are 
CR:  Child  Report 
PCR:  Primary  Caregiver  Report 
TRS:  Teacher  Report  on  Student 
ADHD:  Attention  deficit  hyperactivity  disorder 

The  +/-  signs  in  parentheses  indicate  the  direction  of  a beneficial  outcome.  All  impact  estimates  were  calculated  using  regression  models,  where  each  program  and  school  within  a 
program  was  weighted  equally.  The  standard  errors  of  all  estimates  account  for  design  effects  due  to  unequal  weighting  and  the  clustering  of  students  within  schools. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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Excluding  New  Entrants 

About  7.5  percent  of  children  in  the  spring  2005  follow-up  sample  were  new  entrants  who  enrolled  in  the 
study  schools  after  fall  2004.  For  several  reasons,  the  inclusion  of  these  children  in  the  analysis  sample  could 
have  affected  the  impact  estimates.  First,  new  entrants  were  exposed  to  the  SACD  interventions  for  a shorter 
period  than  were  original  cohort  children,  which  could  have  depressed  the  impact  estimates.  Second,  new 
entrants  may  have  had  different  characteristics  than  did  original  cohort  members,  which  could  have  led  to 
different  program  effects  across  the  two  groups.  For  example,  the  description  of  consent  rates  in  chapter  1 
showed  that  new  entrants  had  lower  consent  rates  (and  percentages  of  the  sample  with  data)  than  did  original 
cohort  members,  potentially  reflecting  such  differences.  Third,  the  new  entrants  in  the  treatment  schools 
could  have  differed  from  those  in  the  control  schools  if  the  SACD  interventions  had  an  effect  on  school 
entry. 15  Finally,  the  imputation  of  baseline  pretest  scores  for  the  new  entrants  (based  on  mean  pretest  scores 
of  original  cohort  members  in  the  same  class)  could  have  influenced  the  impact  estimates. 

As  a sensitivity  test,  models  were  estimated  that  excluded  new  entrants  from  the  analysis  sample  using  Year  1 
and  Year  2 data.  As  with  the  original  impact  estimates,  no  significant  impacts  were  found  when  new  entrants 
were  excluded  (table  B.29). 

For  Years  2 and  3,  the  numbers  of  new  entrants  grew  large  enough  to  do  separate  analyses  of  new  entrants 
and  original  members  of  the  sample.  These  analyses  are  discussed  in  chapter  1 as  part  of  the  subgroup 
analyses.  They  include  the  finding  of  no  significant  impacts  on  the  18  outcomes  for  either  group  and  the 
finding  of  no  significant  differences  between  the  impacts  on  the  new  entrants  versus  the  original  cohort 
members. 


15  Similarly,  the  original  cohort  stayers  in  the  treatment  and  control  group  schools  could  differ  if  the  interventions  had  an 
effect  on  school  exit  rates. 
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Table  B.29.  Combined-program  impacts  in  effect  size  units  when  excluding  new  entrants 


Scale-Report 

Year  1 

(Spring  2005) 

p-value  of 
Impact1  impact2 

Year  2 

(Spring  2006) 

p-value  of 
Impact1  impact2 

Social  and  Emotional  Competence  Domain 

Self-Efficacy  for  Peer  Interactions-CR  (+) 

-0.06 

0.139 

-0.06 

0.176 

Normative  Beliefs  About  Aggression-CR  (-) 

0.00 

0.908 

0.00 

0.996 

Empathy-CR  (+) 

0.06 

0.260 

-0.01 

0.798 

Behavior  Domain 

Altruistic  Behavior-CR  (+) 

-0.07A 

0.080 

-0.03 

0.516 

Altruistic  Behavior-PCR  (+) 

0.05 

0.184 

-0.01 

0.807 

Altruistic  Behavior-TRS  (+) 

0.14 

0.204 

0.02 

0.903 

Positive  Social  Behavior-PCR  (+) 

-0.02 

0.620 

0.01 

0.813 

Positive  Social  Behavior-TRS  (+) 

0.01 

0.822 

-0.03 

0.651 

Problem  Behavior-CR  (-) 

0.01 

0.887 

-0.02 

0.759 

Problem  Behavior-PCR  (+) 

-0.04 

0.411 

-0.05 

0.244 

Problem  Behavior-TRS  (+) 

0.03 

0.438 

-0.03 

0.534 

ADHD-Related  Behavior-TRS  (-) 

0.02 

0.721 

-0.03 

0.590 

Academics  Domain 

Engagement  with  Learning-CR  (+) 

-0.08A 

0.064 

-0.03 

0.533 

Academic  Competence  and  Motivation-TRS  (+) 

-0.07 

0.108 

-0.07 

0.170 

Perceptions  of  School  Climate  Domain 

Positive  School  Orientation-CR  (+) 

0.01 

0.927 

0.05 

0.561 

Negative  School  Orientation-CR  (-) 

-0.01 

0.805 

-0.05 

0.432 

Student  Afraid  at  School-CR  (-) 

-0.06 

0.167 

-0.05 

0.359 

Victimization  at  School-CR  (-) 

-0.01 

0.879 

-0.05 

0.353 

A Significantly  different  from  zero  at  the  .10  to  > .05  level. 

1 1mpacts  are  in  effect  size  (standard  deviation)  units  and  were  calculated  by  dividing  the  estimated  impact  by  the  standard  deviation  of 
the  outcome  measure  for  the  control  group. 

2 The  p-value  is  from  a two-tailed  t test  to  gauge  the  statistical  significance  of  the  impact  estimate. 

NOTE:  Abbreviations  are 
CR:  Child  Report 
PCR:  Primary  Caregiver  Report 
TRS:  Teacher  Report  on  Student 
ADHD:  Attention  deficit  hyperactivity  disorder 

The  + signs  in  parentheses  indicate  the  direction  of  a beneficial  outcome.  No  findings  were  found  statistically  significant  at  or  below  the 
.05  level.  All  impact  estimates  were  calculated  using  regression  models,  where  each  program  and  school  within  a program  was 
weighted  equally.  The  standard  errors  of  all  estimates  account  for  design  effects  due  to  unequal  weighting  and  the  clustering  of 
students  within  schools. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 


Including  Restricted  Sets  of  Covariates  in  the  Regression  Models 

Two  sets  of  regression  models  were  estimated  using  restricted  sets  of  covariates.  In  the  first  set,  the  covariates 
included  only  the  pretest  of  the  outcome  measure.  In  the  second  set,  the  covariates  included  only  the  child 
and  primary  caregiver  demographic  measures  (listed  in  the  first  section  of  this  appendix).  As  with  the  original 
impact  estimates,  no  significant  impacts  were  found  when  using  the  restricted  sets  of  covariates  (table  B.30). 
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Table  B.30.  Combined-program  impacts  in  effect  size  units  using  restricted  sets  of  covariates  (pretest  of  outcomes  only  and  child  and 
primary  caregiver  demographic  measures  only) 


Year  1 

Year  2 

(Spring  2005) 

(Spring  2006) 

Covariates  for 

Covariates  for 

Covariates  for 

Covariates  for 

pretest  of 

child  and 

pretest  of 

child  and 

outcomes  only 

primary  caregiver 

outcomes  only 

primary  caregiver 

p-value  of  p-value  of  p-value  of  p-value  of 


Scale-Report Impact1  impact2  Impact1 impact2  Impact1 impact2  Impact1 impact2 


Social  and  Emotional  Competence  Domain 

Self-Efficacy  for  Peer  Interactions-CR  (+) 

-0.02 

0.520 

-0.05 

0.240 

-0.06 

0.155 

-0.08A 

0.082 

Normative  Beliefs  About  Aggression-CR  (-) 

0.00 

0.911 

-0.02 

0.549 

-0.01 

0.919 

-0.02 

0.656 

Empathy-CR  (+) 

0.06 

0.271 

0.07 

0.144 

-0.03 

0.589 

-0.03 

0.625 

Behavior  Domain 

Altruistic  Behavior-CR  (+) 

-0.07 

0.103 

-0.07 

0.125 

-0.07 

0.125 

-0.07 

0.122 

Altruistic  Behavior-PCR  (+) 

0.06 

0.148 

0.04 

0.307 

0.01 

0.866 

0.00 

0.919 

Altruistic  Behavior-TRS  (+) 

0.07 

0.424 

0.13 

0.236 

0.01 

0.964 

0.02 

0.914 

Positive  Social  Behavior-PCR  (+) 

-0.01 

0.745 

0.01 

0.893 

0.05 

0.230 

0.06 

0.219 

Positive  Social  Behavior-TRS  (+) 

0.01 

0.864 

0.04 

0.568 

-0.05 

0.517 

-0.03 

0.719 

Problem  Behavior-CR  (-) 

0.01 

0.788 

0.00 

0.938 

0.01 

0.848 

0.00 

0.966 

Problem  Behavior-PCR  (+) 

0.01 

0.825 

-0.04 

0.431 

-0.03 

0.470 

-0.08A 

0.062 

Problem  Behavior-TRS  (+) 

0.03 

0.593 

0.00 

0.948 

0.00 

0.937 

-0.02 

0.765 

ADHD-Related  Behavior-TRS  (-) 

0.00 

0.917 

0.00 

0.933 

-0.02 

0.664 

-0.04 

0.661 

Academics  Domain 

Engagement  with  Learning-CR  (+) 

-0.04 

0.267 

-0.05 

0.194 

-0.03 

0.363 

-0.04 

0.246 

Academic  Competence  and  Motivation-TRS  (+) 

-0.01 

0.597 

-0.04 

0.440 

-0.02 

0.701 

-0.05 

0.340 

See  notes  at  end  of  table. 
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Table  B.30.  Combined-program  impacts  in  effect  size  units  using  restricted  sets  of  covariates  (pretest  of  outcomes  only  and  child 
and  primary  caregiver  demographic  measures  only) — Continued 


Year  1 

(Spring  2005) 

Year  2 

(Spring  2006) 

Covariates  for 
pretest  of 
outcomes  only 

Covariates  for 
child  and 
primary  caregiver 

Covariates  for 
pretest  of 
outcomes  only 

Covariates  for 
child  and 
primary  caregiver 

Scale-Report 

Impact1 

p-value  of 
impact2 

Impact1 

p-value  of 
impact2 

Impact1 

p-value  of 
impact2 

Impact1 

p-value  of 
impact2 

Perceptions  of  School  Climate  Domain 
Positive  School  Orientation-CR  (+) 

0.02 

0.747 

0.02 

0.772 

0.02 

0.824 

0.01 

0.887 

Negative  School  Orientation-CR  (-) 

-0.05 

0.354 

-0.04 

0.496 

-0.02 

0.729 

-0.01 

0.868 

Student  Afraid  at  School-CR  (-) 

-0.05 

0.258 

-0.04 

0.397 

-0.05 

0.424 

-0.05 

0.394 

Victimization  at  School-CR  (-) 

-0.02 

0.609 

-0.03 

0.593 

-0.04 

0.420 

-0.05 

0.389 

A Significantly  different  from  zero  at  the  .10  to  > .05  level. 

1 1mpacts  are  in  effect  size  (standard  deviation)  units  and  were  calculated  by  dividing  the  estimated  impact  by  the  standard  deviation  of  the  outcome  measure  for  the  control  group. 
2 The  p-value  is  from  a two-tailed  ttest  to  gauge  the  statistical  significance  of  the  impact  estimate. 

NOTE:  Abbreviations  are 
CR:  Child  Report 
PCR:  Primary  Caregiver  Report 
TRS:  Teacher  Report  on  Student 
ADHD:  Attention  deficit  hyperactivity  disorder 

The  +/-  signs  in  parentheses  indicate  the  direction  of  a beneficial  outcome.  No  findings  were  found  statistically  significant  below  the  .05  level.  All  impact  estimates  were  calculated 
using  regression  models,  where  each  program  and  school  within  a program  was  weighted  equally.  The  standard  errors  of  all  estimates  account  for  design  effects  due  to  unequal 
weighting  and  the  clustering  of  students  within  schools. 

SOURCE:  The  Social  and  Character  Development  (SACD)  Research  Program. 
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